from:"Cabral, Matias A"

Re: [OMPI users] Help Getting Started with Open MPI and PMIx and UCX

2019-01-22 Thread Cabral, Matias A

Hi Matt,

There seem to be two different issues here:

a)  The warning message comes from the openib btl. Given that Omnipath has 
verbs API and you have the necessary libraries in your system, openib btl finds 
itself as a potential transport and prints the warning during its init (openib 
btl is its way to deprecation). You may try to explicitly ask for vader btl 
given you are running on shared mem: -mca btl self,vader -mca pml ob1. Or 
better, explicitly build without openib: ./configure --with-verbs=no …

b)  Not my field of expertise, but you may be having some conflict with the 
external components you are using: --with-pmix=/usr/nlocal/pmix/2.1 
--with-libevent=/usr . You may try not specifying these and using the ones 
provided by OMPI.

_MAC

From: users [mailto:users-boun...@lists.open-mpi.org] On Behalf Of Matt Thompson
Sent: Tuesday, January 22, 2019 6:04 AM
To: Open MPI Users 
Subject: Re: [OMPI users] Help Getting Started with Open MPI and PMIx and UCX

Well,

By turning off UCX compilation per Howard, things get a bit better in that 
something happens! It's not a good something, as it seems to die with an 
infiniband error. As this is an Omnipath system, is OpenMPI perhaps seeing 
libverbs somewhere and compiling it in? To wit:

(1006)(master) $ mpirun -np 4 ./helloWorld.mpi3.SLES12.OMPI400.exe
--
By default, for Open MPI 4.0 and later, infiniband ports on a device
are not used by default.  The intent is to use UCX for these devices.
You can override this policy by setting the btl_openib_allow_ib MCA parameter
to true.

  Local host:  borgc129
  Local adapter:   hfi1_0
  Local port:  1

--
--
WARNING: There was an error initializing an OpenFabrics device.

  Local host:   borgc129
  Local device: hfi1_0
--
Compiler Version: Intel(R) Fortran Intel(R) 64 Compiler for applications 
running on Intel(R) 64, Version 18.0.5.274 Build 20180823
MPI Version: 3.1
MPI Library Version: Open MPI v4.0.0, package: Open MPI mathomp4@discover23 
Distribution, ident: 4.0.0, repo rev: v4.0.0, Nov 12, 2018
[borgc129:260830] *** An error occurred in MPI_Barrier
[borgc129:260830] *** reported by process [140736833716225,46909632806913]
[borgc129:260830] *** on communicator MPI_COMM_WORLD
[borgc129:260830] *** MPI_ERR_OTHER: known error not in list
[borgc129:260830] *** MPI_ERRORS_ARE_FATAL (processes in this communicator will 
now abort,
[borgc129:260830] ***and potentially your MPI job)
forrtl: error (78): process killed (SIGTERM)
Image  PCRoutineLineSource
helloWorld.mpi3.S  0040A38E  for__signal_handl Unknown  Unknown
libpthread-2.22.s  2B9CCB20  Unknown   Unknown  Unknown
libpthread-2.22.s  2B9C90CD  pthread_cond_wait Unknown  Unknown
libpmix.so.2.1.11  2AAAB1D780A1  PMIx_AbortUnknown  Unknown
mca_pmix_ext2x.so  2AAAB1B3AA75  ext2x_abort   Unknown  Unknown
mca_ess_pmi.so 2AAAB1724BC0  Unknown   Unknown  Unknown
libopen-rte.so.40  2C3E941C  orte_errmgr_base_ Unknown  Unknown
mca_errmgr_defaul  2AAABC401668  Unknown   Unknown  Unknown
libmpi.so.40.20.0  2B3CDBC4  ompi_mpi_abortUnknown  Unknown
libmpi.so.40.20.0  2B3BB1EF  ompi_mpi_errors_a Unknown  Unknown
libmpi.so.40.20.0  2B3B99C9  ompi_errhandler_i Unknown  Unknown
libmpi.so.40.20.0  2B3E4576  MPI_Barrier   Unknown  Unknown
libmpi_mpifh.so.4  2B15EE53  MPI_Barrier_f08   Unknown  Unknown
libmpi_usempif08.  2ACE7732  mpi_barrier_f08_  Unknown  Unknown
helloWorld.mpi3.S  0040939F  Unknown   Unknown  Unknown
helloWorld.mpi3.S  0040915E  Unknown   Unknown  Unknown
libc-2.22.so   2BBF96D5  __libc_start_main 
Unknown  Unknown
helloWorld.mpi3.S  00409069  Unknown   Unknown  Unknown

On Sun, Jan 20, 2019 at 4:19 PM Howard Pritchard 
mailto:hpprit...@gmail.com>> wrote:
Hi Matt

Definitely do not include the ucx option for an omnipath cluster.  Actually if 
you accidentally installed ucx in it’s default location use on the system 
Switch to this config option

—with-ucx=no

Otherwise you will hit

https://github.com/openucx/ucx/issues/750

Howard


Gilles Gouaillardet 
mailto:gilles.gouaillar...@gmail.com>> schrieb 
am Sa. 19. Jan. 2019 um 18:41:
Matt,

There are two ways of using PMIx

- if you use mpirun, then the MPI app (e.g. the PMIx client) will talk
to mpirun and orted daemons (e.g. the PMIx server)
- if you use SLURM srun, then the MPI app will directly talk to the
PMIx server provided by SLURM.

Re: [OMPI users] Help Getting Started with Open MPI and PMIx and UCX

2019-01-18 Thread Cabral, Matias A

Hi Matt,

Few comments/questions:

-  If your cluster has Omni-Path, you won’t need UCX. Instead you can 
run using PSM2, or alternatively OFI (a.k.a. Libfabric)

-  With the command you shared below (4 ranks on the local node) (I 
think) a shared mem transport is being selected (vader?). So, if the job is not 
starting this seems to be a runtime issue rather than transport…. Pmix? slurm?
Thanks
_MAC

From: users [mailto:users-boun...@lists.open-mpi.org] On Behalf Of Matt Thompson
Sent: Friday, January 18, 2019 10:27 AM
To: Open MPI Users 
Subject: Re: [OMPI users] Help Getting Started with Open MPI and PMIx and UCX

On Fri, Jan 18, 2019 at 1:13 PM Jeff Squyres (jsquyres) via users 
mailto:users@lists.open-mpi.org>> wrote:
On Jan 18, 2019, at 12:43 PM, Matt Thompson 
mailto:fort...@gmail.com>> wrote:
>
> With some help, I managed to build an Open MPI 4.0.0 with:

We can discuss each of these params to let you know what they are.

> ./configure --disable-wrapper-rpath --disable-wrapper-runpath

Did you have a reason for disabling these?  They're generally good things.  
What they do is add linker flags to the wrapper compilers (i.e., mpicc and 
friends) that basically put a default path to find libraries at run time (that 
can/will in most cases override LD_LIBRARY_PATH -- but you can override these 
linked-in-default-paths if you want/need to).

I've had these in my Open MPI builds for a while now. The reason was one of the 
libraries I need for the climate model I work on went nuts if both of them 
weren't there. It was originally the rpath one but then eventually (Open MPI 
3?) I had to add the runpath one. But I have been updating the libraries more 
aggressively recently (due to OS upgrades) so it's possible this is no longer 
needed.

> --with-psm2

Ensure that Open MPI can include support for the PSM2 library, and abort 
configure if it cannot.

> --with-slurm

Ensure that Open MPI can include support for SLURM, and abort configure if it 
cannot.

> --enable-mpi1-compatibility

Add support for MPI_Address and other MPI-1 functions that have since been 
deleted from the MPI 3.x specification.

> --with-ucx

Ensure that Open MPI can include support for UCX, and abort configure if it 
cannot.

> --with-pmix=/usr/nlocal/pmix/2.1

Tells Open MPI to use the PMIx that is installed at /usr/nlocal/pmix/2.1 
(instead of using the PMIx that is bundled internally to Open MPI's source code 
tree/expanded tarball).

Unless you have a reason to use the external PMIx, the internal/bundled PMIx is 
usually sufficient.

Ah. I did not know that. I figured if our SLURM was built linked to a specific 
PMIx v2 that I should build Open MPI with the same PMIx. I'll build an Open MPI 
4 without specifying this.

> --with-libevent=/usr

Same as previous; change "pmix" to "libevent" (i.e., use the external libevent 
instead of the bundled libevent).

> CC=icc CXX=icpc FC=ifort

Specify the exact compilers to use.

> The MPI 1 is because I need to build HDF5 eventually and I added psm2 because 
> it's an Omnipath cluster. The libevent was probably a red herring as 
> libevent-devel wasn't installed on the system. It was eventually, and I just 
> didn't remove the flag. And I saw no errors in the build!

Might as well remove the --with-libevent if you don't need it.

> However, I seem to have built an Open MPI that doesn't work:
>
> (1099)(master) $ mpirun --version
> mpirun (Open MPI) 4.0.0
>
> Report bugs to http://www.open-mpi.org/community/help/
> (1100)(master) $ mpirun -np 4 ./helloWorld.mpi3.SLES12.OMPI400.exe
>
> It just sits there...forever. Can the gurus here help me figure out what I 
> managed to break? Perhaps I added too much to my configure line? Not enough?

There could be a few things going on here.

Are you running inside a SLURM job?  E.g., in a "salloc" job, or in an "sbatch" 
script?

I have salloc'd 8 nodes of 40 cores each. Intel MPI 18 and 19 work just fine 
(as you'd hope on an Omnipath cluster), but for some reason Open MPI is twitchy 
on this cluster. I once managed to get Open MPI 3.0.1 working (a few months 
ago), and it had some interesting startup scaling I liked (slow at low core 
count, but getting close to Intel MPI at high core count), though it seemed to 
not work after about 100 nodes (4000 processes) or so.

--
Matt Thompson
   “The fact is, this is about us identifying what we do best and
   finding more ways of doing less of it better” -- Director of Better Anna 
Rampton
___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users

Re: [OMPI users] Open MPI 4.0.0 - error with MPI_Send

2019-01-16 Thread Cabral, Matias A

Hey Eduardo,

Using up to date libraries is advisable, especially given that 1.4.0 is a 
couple years old.  1.6.2 is the latest on the 1.6.x line. 1.7.0 was released 
last week, however I have not played with it yet.

Thanks
_MAC

From: users [mailto:users-boun...@lists.open-mpi.org] On Behalf Of ROTHE 
Eduardo - externe
Sent: Wednesday, January 16, 2019 9:29 AM
To: Open MPI Users 
Subject: Re: [OMPI users] Open MPI 4.0.0 - error with MPI_Send


Hi Matias,



thanks so much for your support!



Actually running this simple example with --mca mtl_ofi_tag_mode ofi_tag_1  
turns out to be a good choice! I mean, the following execution do  not return 
the MPI_Send error any more:



mpirun -np 2 --mca mtl_ofi_tag_mode ofi_tag_1 ./a



Are you suggesting that upgrading  libfabric   to  1.6.0   might save the day?



Regards,

Eduardo




De : users 
mailto:users-boun...@lists.open-mpi.org>> de 
la part de matias.a.cab...@intel.com<mailto:matias.a.cab...@intel.com> 
mailto:matias.a.cab...@intel.com>>
Envoyé : mercredi 16 janvier 2019 00:54
À : Open MPI Users
Objet : Re: [OMPI users] Open MPI 4.0.0 - error with MPI_Send

Hi Eduardo,


> When you say that "The OFI MTL got some new features during 2018 that went 
> into v4.0.0 but are not backported to older OMPI versions." this agrees with 
> the bahaviour that I witness - using Open MPI 3.1.3 I don't get this error. 
> Could this be related?

Yes. I suspect this may be related to the inclusion of support for 
FI_REMOTE_CQ_DATA that was added to extend scalability of OFI MTL. This went 
into 4.x, but is not in 3.1.x. In addition there is a bug in the PSM2 OFI 
provider that reports support for FI_REMOTE_CQ_DATA when it actually did not 
support the API that OMPI uses (this was fixed in libfabric 1.6.0). A quick way 
to test this would be adding  '-mca mtl_ofi_tag_mode ofi_tag_1' to your command 
line. This would force OMPI not using FI_REMOTE_CQ_DATA.

Thanks,

From: users [mailto:users-boun...@lists.open-mpi.org] On Behalf Of ROTHE 
Eduardo - externe
Sent: Tuesday, January 15, 2019 2:31 AM
To: Open MPI Users mailto:users@lists.open-mpi.org>>
Subject: Re: [OMPI users] Open MPI 4.0.0 - error with MPI_Send


Hi Matias,



Thank you so much for your feedback!



It's really embarrassing, but running



mpirun -np 2 -mca mtl ofi -mca pml cm -mca mtl_ofi_provider_include psm2 ./a



still doesn't get the job done. I'm still getting the same MPI_Send error:



Hello World from proccess 1 out of 2
Hello World from proccess 0 out of 2
[gafront4:18272] *** An error occurred in MPI_Send
[gafront4:18272] *** reported by process [2565799937,0]
[gafront4:18272] *** on communicator MPI_COMM_WORLD
[gafront4:18272] *** MPI_ERR_OTHER: known error not in list
[gafront4:18272] *** MPI_ERRORS_ARE_FATAL (processes in this communicator 
will now abort,
[gafront4:18272] ***and potentially your MPI job)



I'm using libfabric-1.4.0  issued from Debian Stretch with a minor modification 
to use PSM2. It can be found here:



https://github.com/scibian/libfabric/commits/scibian/opa10.7/stretch



When you say that "The OFI MTL got some new features during 2018 that went into 
v4.0.0 but are not backported to older OMPI versions." this agrees with the 
bahaviour that I witness - using Open MPI 3.1.3 I don't get this error. Could 
this be related?



Regards,

Eduardo




De : users 
mailto:users-boun...@lists.open-mpi.org>> de 
la part de matias.a.cab...@intel.com<mailto:matias.a.cab...@intel.com> 
mailto:matias.a.cab...@intel.com>>
Envoyé : samedi 12 janvier 2019 00:32
À : Open MPI Users
Objet : Re: [OMPI users] Open MPI 4.0.0 - error with MPI_Send

BTW, just to be explicit about using the psm2 OFI provider:

/tmp> mpirun -np 2 -mca mtl ofi -mca pml cm -mca mtl_ofi_provider_include psm2 
./a
Hello World from proccess 0 out of 2
This is process 0 reporting::
Hello World from proccess 1 out of 2
Process 1 received number 10 from process 0

From: users [mailto:users-boun...@lists.open-mpi.org] On Behalf Of Cabral, 
Matias A
Sent: Friday, January 11, 2019 3:22 PM
To: Open MPI Users mailto:users@lists.open-mpi.org>>
Subject: Re: [OMPI users] Open MPI 4.0.0 - error with MPI_Send

Hi Eduardo,

The OFI MTL got some new features during 2018 that went into v4.0.0 but are not 
backported to older OMPI versions.

What version of libfabric are you using and where are you installing it from?  
I will try to reproduce your error. I'm running some quick tests and I see it 
working:



/tmp >ompi_info
 Package: Open MPI 
macab...@sperf-41.sc.intel.com<mailto:macab...@sperf-41.sc.intel.com>
  Distribution
Open MPI: 4.0.0rc5
  Open MPI repo revision: v4.0.0
   Open MPI release date: Unreleased developer copy
Open RTE: 4.0.0rc5
  Open RTE repo r

Re: [OMPI users] Open MPI 4.0.0 - error with MPI_Send

2019-01-15 Thread Cabral, Matias A

Hi Eduardo,


> When you say that "The OFI MTL got some new features during 2018 that went 
> into v4.0.0 but are not backported to older OMPI versions." this agrees with 
> the bahaviour that I witness - using Open MPI 3.1.3 I don't get this error. 
> Could this be related?

Yes. I suspect this may be related to the inclusion of support for 
FI_REMOTE_CQ_DATA that was added to extend scalability of OFI MTL. This went 
into 4.x, but is not in 3.1.x. In addition there is a bug in the PSM2 OFI 
provider that reports support for FI_REMOTE_CQ_DATA when it actually did not 
support the API that OMPI uses (this was fixed in libfabric 1.6.0). A quick way 
to test this would be adding  '-mca mtl_ofi_tag_mode ofi_tag_1' to your command 
line. This would force OMPI not using FI_REMOTE_CQ_DATA.

Thanks,

From: users [mailto:users-boun...@lists.open-mpi.org] On Behalf Of ROTHE 
Eduardo - externe
Sent: Tuesday, January 15, 2019 2:31 AM
To: Open MPI Users 
Subject: Re: [OMPI users] Open MPI 4.0.0 - error with MPI_Send


Hi Matias,



Thank you so much for your feedback!



It's really embarrassing, but running



mpirun -np 2 -mca mtl ofi -mca pml cm -mca mtl_ofi_provider_include psm2 ./a



still doesn't get the job done. I'm still getting the same MPI_Send error:



Hello World from proccess 1 out of 2
Hello World from proccess 0 out of 2
[gafront4:18272] *** An error occurred in MPI_Send
[gafront4:18272] *** reported by process [2565799937,0]
[gafront4:18272] *** on communicator MPI_COMM_WORLD
[gafront4:18272] *** MPI_ERR_OTHER: known error not in list
[gafront4:18272] *** MPI_ERRORS_ARE_FATAL (processes in this communicator 
will now abort,
[gafront4:18272] ***and potentially your MPI job)



I'm using libfabric-1.4.0  issued from Debian Stretch with a minor modification 
to use PSM2. It can be found here:



https://github.com/scibian/libfabric/commits/scibian/opa10.7/stretch



When you say that "The OFI MTL got some new features during 2018 that went into 
v4.0.0 but are not backported to older OMPI versions." this agrees with the 
bahaviour that I witness - using Open MPI 3.1.3 I don't get this error. Could 
this be related?



Regards,

Eduardo




De : users 
mailto:users-boun...@lists.open-mpi.org>> de 
la part de matias.a.cab...@intel.com<mailto:matias.a.cab...@intel.com> 
mailto:matias.a.cab...@intel.com>>
Envoyé : samedi 12 janvier 2019 00:32
À : Open MPI Users
Objet : Re: [OMPI users] Open MPI 4.0.0 - error with MPI_Send

BTW, just to be explicit about using the psm2 OFI provider:

/tmp> mpirun -np 2 -mca mtl ofi -mca pml cm -mca mtl_ofi_provider_include psm2 
./a
Hello World from proccess 0 out of 2
This is process 0 reporting::
Hello World from proccess 1 out of 2
Process 1 received number 10 from process 0

From: users [mailto:users-boun...@lists.open-mpi.org] On Behalf Of Cabral, 
Matias A
Sent: Friday, January 11, 2019 3:22 PM
To: Open MPI Users mailto:users@lists.open-mpi.org>>
Subject: Re: [OMPI users] Open MPI 4.0.0 - error with MPI_Send

Hi Eduardo,

The OFI MTL got some new features during 2018 that went into v4.0.0 but are not 
backported to older OMPI versions.

What version of libfabric are you using and where are you installing it from?  
I will try to reproduce your error. I'm running some quick tests and I see it 
working:



/tmp >ompi_info
 Package: Open MPI 
macab...@sperf-41.sc.intel.com<mailto:macab...@sperf-41.sc.intel.com>
  Distribution
Open MPI: 4.0.0rc5
  Open MPI repo revision: v4.0.0
   Open MPI release date: Unreleased developer copy
Open RTE: 4.0.0rc5
  Open RTE repo revision: v4.0.0
   Open RTE release date: Unreleased developer copy
OPAL: 4.0.0rc5
  OPAL repo revision: v4.0.0
   OPAL release date: Unreleased developer copy
 MPI API: 3.1.0
Ident string: 4.0.0rc5
  Prefix: /nfs/sc/disks/fabric_work/macabral/tmp/ompi-4.0.0
Configured architecture: x86_64-unknown-linux-gnu
  Configure host: sperf-41.sc.intel.com
   Configured by: macabral
   Configured on: Fri Jan 11 17:42:06 EST 2019
  Configure host: sperf-41.sc.intel.com
  Configure command line: '--with-ofi' '--with-verbs=no'
  '--prefix=/tmp/ompi-4.0.0'

/tmp> rpm -qi libfabric
Name: libfabric
Version : 1.6.0
Release : 80
Architecture: x86_64
Install Date: Wed 19 Dec 2018 05:45:41 PM EST
Group   : System Environment/Libraries
Size: 10131964
License : GPLv2 or BSD
Signature   : (none)
Source RPM  : libfabric-1.6.0-80.src.rpm
Build Date  : Wed 22 Aug 2018 11:08:29 PM EDT
Build Host  : ph-bld-node-27.ph.intel.com
Relocations : (not relocatable)
URL : http://www.github.com/ofiwg/libfabric
Summary : User-space RDMA Fabric Interfaces
Description

Re: [OMPI users] Open MPI 4.0.0 - error with MPI_Send

2019-01-11 Thread Cabral, Matias A

BTW, just to be explicit about using the psm2 OFI provider:

/tmp> mpirun -np 2 -mca mtl ofi -mca pml cm -mca mtl_ofi_provider_include psm2 
./a
Hello World from proccess 0 out of 2
This is process 0 reporting::
Hello World from proccess 1 out of 2
Process 1 received number 10 from process 0

From: users [mailto:users-boun...@lists.open-mpi.org] On Behalf Of Cabral, 
Matias A
Sent: Friday, January 11, 2019 3:22 PM
To: Open MPI Users 
Subject: Re: [OMPI users] Open MPI 4.0.0 - error with MPI_Send

Hi Eduardo,

The OFI MTL got some new features during 2018 that went into v4.0.0 but are not 
backported to older OMPI versions.

What version of libfabric are you using and where are you installing it from?  
I will try to reproduce your error. I'm running some quick tests and I see it 
working:



/tmp >ompi_info
 Package: Open MPI 
macab...@sperf-41.sc.intel.com<mailto:macab...@sperf-41.sc.intel.com>
  Distribution
Open MPI: 4.0.0rc5
  Open MPI repo revision: v4.0.0
   Open MPI release date: Unreleased developer copy
Open RTE: 4.0.0rc5
  Open RTE repo revision: v4.0.0
   Open RTE release date: Unreleased developer copy
OPAL: 4.0.0rc5
  OPAL repo revision: v4.0.0
   OPAL release date: Unreleased developer copy
 MPI API: 3.1.0
Ident string: 4.0.0rc5
  Prefix: /nfs/sc/disks/fabric_work/macabral/tmp/ompi-4.0.0
Configured architecture: x86_64-unknown-linux-gnu
  Configure host: sperf-41.sc.intel.com
   Configured by: macabral
   Configured on: Fri Jan 11 17:42:06 EST 2019
  Configure host: sperf-41.sc.intel.com
  Configure command line: '--with-ofi' '--with-verbs=no'
  '--prefix=/tmp/ompi-4.0.0'

/tmp> rpm -qi libfabric
Name: libfabric
Version : 1.6.0
Release : 80
Architecture: x86_64
Install Date: Wed 19 Dec 2018 05:45:41 PM EST
Group   : System Environment/Libraries
Size: 10131964
License : GPLv2 or BSD
Signature   : (none)
Source RPM  : libfabric-1.6.0-80.src.rpm
Build Date  : Wed 22 Aug 2018 11:08:29 PM EDT
Build Host  : ph-bld-node-27.ph.intel.com
Relocations : (not relocatable)
URL : http://www.github.com/ofiwg/libfabric
Summary : User-space RDMA Fabric Interfaces
Description :
libfabric provides a user-space API to access high-performance fabric
services, such as RDMA.

/tmp> mpirun -np 2 -mca mtl ofi -mca pml cm ./a
Hello World from proccess 0 out of 2
This is process 0 reporting::
Hello World from proccess 1 out of 2
Process 1 received number 10 from process 0


From: users [mailto:users-boun...@lists.open-mpi.org] On Behalf Of ROTHE 
Eduardo - externe
Sent: Thursday, January 10, 2019 10:02 AM
To: Open MPI Users mailto:users@lists.open-mpi.org>>
Subject: Re: [OMPI users] Open MPI 4.0.0 - error with MPI_Send


Hi Gilles, thank you so much once again!

I have a success using directly the psm2 mtl. Indeed, I do not need to use the 
cm pml (I guess this might be because the cm pml gets automatically selected 
when I enforce the psm2 mtl?). So both the following two commands execute 
successfully with Open MPI 4.0.0:

  > mpirun --mca pml cm --mca mtl psm2 -np 2 ./a.out
  > mpirun --mca mtl psm2 -np 2 ./a.out

The error persists using libfabric. The following command returns the MPI_Send 
error:

  > mpirun --mca pml cm --mca mtl ofi -np 2 ./a.out

It seems the problem sits between libfabric and Open MPI 4.0.0 (remember, I 
don't see the same behaviour with Open MPI 3.1.3). So I guess if I want to use 
libfabric I will have to dig a bit more regarding the interface between this 
library and Open MPI 4.0.0. Any lines of thought on how to start here would be 
(very!) appreciated.

If you have any scheme that would help me to understand the framework/modules 
architecture and why some modules are automatically selected (like the above 
case), I would be very pleased (even more!?:).

Regards,
Eduardo




De : users 
mailto:users-boun...@lists.open-mpi.org>> de 
la part de gilles.gouaillar...@gmail.com<mailto:gilles.gouaillar...@gmail.com> 
mailto:gilles.gouaillar...@gmail.com>>
Envoyé : jeudi 10 janvier 2019 13:51
À : Open MPI Users
Objet : Re: [OMPI users] Open MPI 4.0.0 - error with MPI_Send

Eduardo,

You have two options to use OmniPath

- "directly" via the psm2 mtl
mpirun -mca pml cm -mca mtl psm2 ...

- "indirectly" via libfabric
mpirun -mca pml cm -mca mtl ofi ...

I do invite you to try both. By explicitly requesting the mtl you will avoid 
potential conflicts.

libfabric is used in production by Cisco and AWS (both major contributors to 
both Open MPI and libfabric) so this is clearly not something to stay away 
from. That being said, bug always happen and they could be related to Open MPI, 
libfabric and/or OmniPath (and fwiw, Intel is a major contributor to

Re: [OMPI users] Open MPI 4.0.0 - error with MPI_Send

2019-01-11 Thread Cabral, Matias A

Hi Eduardo,

The OFI MTL got some new features during 2018 that went into v4.0.0 but are not 
backported to older OMPI versions.

What version of libfabric are you using and where are you installing it from?  
I will try to reproduce your error. I'm running some quick tests and I see it 
working:

/tmp >ompi_info
 Package: Open MPI macab...@sperf-41.sc.intel.com
  Distribution
Open MPI: 4.0.0rc5
  Open MPI repo revision: v4.0.0
   Open MPI release date: Unreleased developer copy
Open RTE: 4.0.0rc5
  Open RTE repo revision: v4.0.0
   Open RTE release date: Unreleased developer copy
OPAL: 4.0.0rc5
  OPAL repo revision: v4.0.0
   OPAL release date: Unreleased developer copy
 MPI API: 3.1.0
Ident string: 4.0.0rc5
  Prefix: /nfs/sc/disks/fabric_work/macabral/tmp/ompi-4.0.0
Configured architecture: x86_64-unknown-linux-gnu
  Configure host: sperf-41.sc.intel.com
   Configured by: macabral
   Configured on: Fri Jan 11 17:42:06 EST 2019
  Configure host: sperf-41.sc.intel.com
  Configure command line: '--with-ofi' '--with-verbs=no'
  '--prefix=/tmp/ompi-4.0.0'

/tmp> rpm -qi libfabric
Name: libfabric
Version : 1.6.0
Release : 80
Architecture: x86_64
Install Date: Wed 19 Dec 2018 05:45:41 PM EST
Group   : System Environment/Libraries
Size: 10131964
License : GPLv2 or BSD
Signature   : (none)
Source RPM  : libfabric-1.6.0-80.src.rpm
Build Date  : Wed 22 Aug 2018 11:08:29 PM EDT
Build Host  : ph-bld-node-27.ph.intel.com
Relocations : (not relocatable)
URL : http://www.github.com/ofiwg/libfabric
Summary : User-space RDMA Fabric Interfaces
Description :
libfabric provides a user-space API to access high-performance fabric
services, such as RDMA.

/tmp> mpirun -np 2 -mca mtl ofi -mca pml cm ./a
Hello World from proccess 0 out of 2
This is process 0 reporting::
Hello World from proccess 1 out of 2
Process 1 received number 10 from process 0

From: users [mailto:users-boun...@lists.open-mpi.org] On Behalf Of ROTHE 
Eduardo - externe
Sent: Thursday, January 10, 2019 10:02 AM
To: Open MPI Users 
Subject: Re: [OMPI users] Open MPI 4.0.0 - error with MPI_Send

Hi Gilles, thank you so much once again!

I have a success using directly the psm2 mtl. Indeed, I do not need to use the 
cm pml (I guess this might be because the cm pml gets automatically selected 
when I enforce the psm2 mtl?). So both the following two commands execute 
successfully with Open MPI 4.0.0:

  > mpirun --mca pml cm --mca mtl psm2 -np 2 ./a.out
  > mpirun --mca mtl psm2 -np 2 ./a.out

The error persists using libfabric. The following command returns the MPI_Send 
error:

  > mpirun --mca pml cm --mca mtl ofi -np 2 ./a.out

It seems the problem sits between libfabric and Open MPI 4.0.0 (remember, I 
don't see the same behaviour with Open MPI 3.1.3). So I guess if I want to use 
libfabric I will have to dig a bit more regarding the interface between this 
library and Open MPI 4.0.0. Any lines of thought on how to start here would be 
(very!) appreciated.

If you have any scheme that would help me to understand the framework/modules 
architecture and why some modules are automatically selected (like the above 
case), I would be very pleased (even more!?:).

Regards,
Eduardo

De : users 
mailto:users-boun...@lists.open-mpi.org>> de 
la part de gilles.gouaillar...@gmail.com 
mailto:gilles.gouaillar...@gmail.com>>
Envoyé : jeudi 10 janvier 2019 13:51
À : Open MPI Users
Objet : Re: [OMPI users] Open MPI 4.0.0 - error with MPI_Send

Eduardo,

You have two options to use OmniPath

- "directly" via the psm2 mtl
mpirun -mca pml cm -mca mtl psm2 ...

- "indirectly" via libfabric
mpirun -mca pml cm -mca mtl ofi ...

I do invite you to try both. By explicitly requesting the mtl you will avoid 
potential conflicts.

libfabric is used in production by Cisco and AWS (both major contributors to 
both Open MPI and libfabric) so this is clearly not something to stay away 
from. That being said, bug always happen and they could be related to Open MPI, 
libfabric and/or OmniPath (and fwiw, Intel is a major contributor to libfabric 
too)

Cheers,

Gilles

On Thursday, January 10, 2019, ROTHE Eduardo - externe 
mailto:eduardo-externe.ro...@edf.fr>> wrote:

Hi Gilles, thank you so much for your support!

For now I'm just testing the software, so it's running on a single node.

Your suggestion was very precise. In fact, choosing the ob1 component leads to 
a successfull execution! The tcp component had no effect.

mpirun --mca pml ob1 -mca btl tcp,self -np 2 ./a.out > Success
mpirun --mca pml ob1 -np 2 ./a.out > Success

But... our cluster is equiped with Intel OMNI Path interconnects and we are 
aiming to use psm2 through ofi component in order to take full

Re: [OMPI users] A couple of general questions

2018-06-14 Thread Cabral, Matias A

Hey Jeff, 

I will help with the OFI part. 

Thanks,
_MAC

-Original Message-
From: users [mailto:users-boun...@lists.open-mpi.org] On Behalf Of Jeff Squyres 
(jsquyres) via users
Sent: Thursday, June 14, 2018 12:50 PM
To: Open MPI User's List 
Cc: Jeff Squyres (jsquyres) 
Subject: Re: [OMPI users] A couple of general questions

Yeah, keeping the documentation / FAQ up to date is... difficult.  :-(

We could definitely use some help with that.

Does anyone have some cycles to help update our FAQ, perchance?


> On Jun 14, 2018, at 11:08 AM, Charles A Taylor  wrote:
> 
> Thank you, Jeff.
> 
> The ofi MTL with the verbs provider seems to be working well at the moment.  
> I’ll need to let it run a day or so before I know whether we can avoid the 
> deadlocks experienced with the straight openib BTL.
> 
> I’ve also built-in UCX support so I’ll be trying that next.  
> 
> Again, thanks for the response.
> 
> Oh, before I forget and I hope this doesn’t sound snarky, but how does the 
> community find out that things like UCX and libfabric exist as well as how to 
> use them when the FAQs on open-mpi.org don’t have much information beyond the 
> now ancient 1.8 series?   Afterall, this is hardly your typical “mpiexec” 
> command line…
> 
>  mpirun -mca pml cm -mca mtl ofi -mca mtl_ofi_provider_include 
> “verbs;ofi_rxm ...”  ,
> 
> if you get my drift.  Even google doesn’t seem to know all that much about 
> these things.  I’m feeling more than a little ignorant these days.  :)
> 
> Thanks to all for the responses.  It has been a huge help.
> 
> Charlie
> 
>> On Jun 14, 2018, at 1:18 PM, Jeff Squyres (jsquyres) via users 
>>  wrote:
>> 
>> Charles --
>> 
>> It may have gotten lost in the middle of this thread, but the 
>> vendor-recommended way of running on InfiniBand these days is with UCX.  
>> I.e., install OpenUCX and use one of the UCX transports in Open MPI.  Unless 
>> you have special requirements, you should likely give this a try and see if 
>> it works for you.
>> 
>> The libfabric / verbs combo *may* work, but I don't know how robust the 
>> verbs libfabric support was in the v1.5 release series.
> 
> ___
> users mailing list
> users@lists.open-mpi.org
> https://lists.open-mpi.org/mailman/listinfo/users


-- 
Jeff Squyres
jsquy...@cisco.com

___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users
___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users

Re: [OMPI users] A couple of general questions

2018-06-14 Thread Cabral, Matias A

Hi Charles,

What version of libfabric do you have installed? To run OMPI using the verbs 
provider you need to pair it with the ofi_rxm provider. fi_info should list it 
like:
…
provider: verbs;ofi_rxm
…

So in your command line you have to specify:
mpirun -mca pml cm -mca mtl ofi -mca mtl_ofi_provider_include “verbs;ofi_rxm”  
….

(don’t skip the quotes)

> Unfortunately, the openmpi-org website FAQ’s covering OpenFabrics support 
> don’t mention anything beyond OpenMPI 1.8.
Good feedback, I’ll look to see how this could be improved.

Thanks,

_MAC

From: users [mailto:users-boun...@lists.open-mpi.org] On Behalf Of Charles A 
Taylor
Sent: Thursday, June 14, 2018 6:09 AM
To: Open MPI Users 
Subject: Re: [OMPI users] A couple of general questions

FYI…

GIZMO: prov/verbs/src/ep_rdm/verbs_tagged_ep_rdm.c:443: 
fi_ibv_rdm_tagged_release_remote_sbuff: Assertion `0' failed.

GIZMO:10405 terminated with signal 6 at PC=2add5835c1f7 SP=7fff8071b008.  
Backtrace:
/usr/lib64/libc.so.6(gsignal+0x37)[0x2add5835c1f7]
/usr/lib64/libc.so.6(abort+0x148)[0x2add5835d8e8]
/usr/lib64/libc.so.6(+0x2e266)[0x2add58355266]
/usr/lib64/libc.so.6(+0x2e312)[0x2add58355312]
/lib64/libfabric.so.1(+0x4df43)[0x2add5b87df43]
/lib64/libfabric.so.1(+0x43af2)[0x2add5b873af2]
/lib64/libfabric.so.1(+0x43ea9)[0x2add5b873ea9]

On Jun 14, 2018, at 7:48 AM, Howard Pritchard 
mailto:hpprit...@gmail.com>> wrote:

Hello Charles

You are heading in the right direction.

First you might want to run the libfabric fi_info command to see what 
capabilities you picked up from the libfabric RPMs.

Next you may well not actually be using the OFI  mtl.

Could you run your app with

export OMPI_MCA_mtl_base_verbose=100

and post the output?

It would also help if you described the system you are using :  OS interconnect 
cpu type etc.

Howard

Charles A Taylor mailto:chas...@ufl.edu>> schrieb am Do. 14. 
Juni 2018 um 06:36:
Because of the issues we are having with OpenMPI and the openib BTL (questions 
previously asked), I’ve been looking into what other transports are available.  
I was particularly interested in OFI/libfabric support but cannot find any 
information on it more recent than a reference to the usNIC BTL from 2015 (Jeff 
Squyres, Cisco).  Unfortunately, the openmpi-org website FAQ’s covering 
OpenFabrics support don’t mention anything beyond OpenMPI 1.8.  Given that 3.1 
is the current stable version, that seems odd.

That being the case, I thought I’d ask here. After laying down the 
libfabric-devel RPM and building (3.1.0) with —with-libfabric=/usr, I end up 
with an “ofi” MTL but nothing else.   I can run with OMPI_MCA_mtl=ofi and 
OMPI_MCA_btl=“self,vader,openib” but it eventually crashes in libopen-pal.so.   
(mpi_waitall() higher up the stack).

GIZMO:9185 terminated with signal 11 at PC=2b4d4b68a91d SP=7ffcfbde9ff0.  
Backtrace:
/apps/mpi/intel/2018.1.163/openmpi/3.1.0/lib64/libopen-pal.so.40(+0x9391d)[0x2b4d4b68a91d]
/apps/mpi/intel/2018.1.163/openmpi/3.1.0/lib64/libopen-pal.so.40(opal_progress+0x24)[0x2b4d4b632754]
/apps/mpi/intel/2018.1.163/openmpi/3.1.0/lib64/libmpi.so.40(ompi_request_default_wait_all+0x11f)[0x2b4d47be2a6f]
/apps/mpi/intel/2018.1.163/openmpi/3.1.0/lib64/libmpi.so.40(PMPI_Waitall+0xbd)[0x2b4d47c2ce4d]

Questions: Am I using the OFI MTL as intended?   Should there be an “ofi” BTL?  
 Does anyone use this?

Thanks,

Charlie Taylor
UF Research Computing

PS - If you could use some help updating the FAQs, I’d be willing to put in 
some time.  I’d probably learn a lot.
___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users
___
users mailing list
users@lists.open-mpi.org
https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.open-2Dmpi.org_mailman_listinfo_users=DwICAg=pZJPUDQ3SB9JplYbifm4nt2lEVG5pWx2KikqINpWlZM=HOtXciFqK5GlgIgLAxthUQ=nOFQDWuhmU9qhe6be-0JeNMGn1q64kJj0nWQV-vZg7k=PoOVfxkE7rR9spMSFabAs8TokTpgbCIyJRGuWTf5jIk=

___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users

Re: [OMPI users] OpenMPI with PSM on True Scale with OmniPath drivers

2018-01-22 Thread Cabral, Matias A

Hi William, 

Couple other questions: 
 - Please share how you ompi configure line looks like. 
-  Please clarify which is/are the compat libraries you refer to. There are 
some that are actually for the opposite case: Making TS app/libs run on 
Omnipath. 
-  As Gilles mentions, moving to a newer major OMPI version is advisable. If 
this is not possible, move to 1.10.7 that has many updates against 1.10.1. 

Thanks, 

_MAC


-Original Message-
From: users [mailto:users-boun...@lists.open-mpi.org] On Behalf Of Gilles 
Gouaillardet
Sent: Monday, January 22, 2018 3:31 AM
To: Open MPI Users 
Subject: Re: [OMPI users] OpenMPI with PSM on True Scale with OmniPath drivers

William,

In order to force PSM (aka Infinipath) you can

mpirun --mca pml cm --mca mtl psm ...

(Replace with psm2 for PSM2 (aka Omnipath)

You can also

mpirun --mca pml_base_verbose 10 --mca mtl_base_verbose 10 ...

in order to collect some logs.

Bottom line, pml/cm should be selected (instead of pml/ob1) and the appropriate 
mtl should be selected.


On top of that, you might need to rebuild Open MPI if some user level library 
has been changed.

Note Open MPI 1.10 is now legacy, and I strongly encourage you to upgrade to 
2.1.x or 3.0.x


Cheers,

Gilles


William Hay  wrote:
>We have a couple of clusters with Qlogic Infinipath/Intel TrueScale 
>networking.  While testing a kernel upgrade we find that the Truescale 
>drivers will no longer build against recent RHEL kernels.  Intel tells 
>us that the Omnipath drivers will work for True Scale adapters so we 
>install those.  Basic functionality appears fine however we are having 
>trouble getting OpenMPI to work.
>
>Using our existing builds of OpenMPI 1.10 jobs receive lots of signal
>11 and crash(output attached)
>
>If we modify LD_LIBRARY_PATH to point to the directory containing the 
>compatibility library provides as part of the OmniPath drivers it 
>instead produces complainst about not finding /dev/hfi1_0 which exists 
>on our cluster with actual OmniPath but not on the clusters with 
>TrueScale (output also attached).
>
>We had a similar issue with Intel MPI but there it was possible to get 
>it to work by passing a -psm option to mpirun.  That combined with the 
>mention of PSM2 in the output when complaining about /dev/hfi1_0 makes 
>us think OpenMPI is trying to run with PSM2 rather than the original 
>PSM and failing because that isn't supported by TrueScale.
>
>We hoped that there would be an mca parameter or combination of 
>parameters that would resolve this issue but while Googling has turned 
>up a few things that look like they would force the use of PSM over 
>PSM2 none of them seem to make a difference.
>
>Any suggestions?
>
>William
___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users
___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users

Re: [OMPI users] Received eager message(s) from an unknown process error on KNL

2017-04-28 Thread Cabral, Matias A

Hi Esthela,

As George mentions, this is indeed libpsm2 printing this error. Opcode=0xCC is 
a disconnect retry. There are a few scenarios that could be happening, but can 
simplify in saying it is an already disconnected endpoint message arriving 
late. What version of Intel Ompin-path Software or libpsm2 do you have in your 
system? We have not seen this error since the release of IFS 10.3.0. I suggest 
updating and testing again.

https://downloadcenter.intel.com/download/26567/Intel-Omni-Path-Fabric-Software-Including-Intel-Omni-Path-Host-Fabric-Interface-Driver-?v=t

Thanks,

_MAC

From: users [mailto:users-boun...@lists.open-mpi.org] On Behalf Of George 
Bosilca
Sent: Thursday, April 27, 2017 7:46 PM
To: Open MPI Users 
Subject: Re: [OMPI users] Received eager message(s) from an unknown process 
error on KNL

Esthela,

This error message is generated internally by the PSM2 library, so you will not 
be able to get rid of it simply by recompiling Open MPI.

  George.


On Thu, Apr 27, 2017 at 8:21 PM, Gallardo, Esthela 
> wrote:
Hello,

I am currently running a couple of benchmarks on two Intel Xeon Phi 7250 
second-generation KNL MIC compute nodes using Open MPI 2.1.0. While trying to 
run the osu_bcast benchmark with 8 MPI tasks (4 on each node), I noticed the 
following error in my output:

Received eager message(s) ptype=0x1 opcode=0xcc from an unknown process (err=49)

I have tried running the benchmark in the following manners:
mpirun -np 8 ./osu_bcast
mpirun -np 8 -hostfile hosti --npernode 4  ./osu_bcast
mpirun -np 8 -hostfile hosti --npernode 4 --mca mtl psm2  ./osu_bcast

But, nothing changes the error message at the end. Note, that the error does 
not really impact the results of the benchmark, so it’s possible that the error 
may be occurring in MPI_Finalize.

Also, in order to try to avoid getting this error, I tried to build the library 
with both of these configurations:
 ./configure --prefix= CC=icc CXX=icpc FC=ifort 
CFLAGS=-xCORE-AVX2 -axMIC-AVX512 CXXFLAGS=-xCORE-AVX2 -axMIC-AVX512 
FFLAGS=-xCORE-AVX2 -axMIC-AVX512 LDFLAGS=-xCORE-AVX2 -axMIC-AVX512

 ./configure --prefix= —enable-orterun-prefix-by-default 
—with-cma=yes --with-psm2 CC=icc CXX=icpc FC=ifort --disable-shared 
--enable-static  --without-slurm

However, this did not help prevent the occurrence of the error either. I was 
wondering if anyone has encountered this issue before, and what can be done in 
order to get rid of the error message.

Thank you,

Esthela Gallardo



___
users mailing list
users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

___
users mailing list
users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

Re: [OMPI users] openmpi single node jobs using btl openib

2017-02-08 Thread Cabral, Matias A

Hi Jingchao,

The log shows the psm mtl is being selected.
…
[c1725.crane.hcc.unl.edu:187002] select: init returned priority 20
[c1725.crane.hcc.unl.edu:187002] selected cm best priority 30
[c1725.crane.hcc.unl.edu:187002] select: component ob1 not selected / finalized
[c1725.crane.hcc.unl.edu:187002] select: component cm selected
…
[c1725.crane.hcc.unl.edu:187002] mca:base:select: Auto-selecting mtl components
[c1725.crane.hcc.unl.edu:187002] mca:base:select:(  mtl) Querying component 
[psm]
[c1725.crane.hcc.unl.edu:187002] mca:base:select:(  mtl) Query of component 
[psm] set priority to 30
[c1725.crane.hcc.unl.edu:187002] mca:base:select:(  mtl) Selected component 
[psm]
[c1725.crane.hcc.unl.edu:187002] select: initializing mtl component psm
…

So, I suspect you may be seeing the same issues that Hristo also reported in 
“the other email thread” when running CP2K and using PSM shm device.  
Therefore, if you are running in a single node, you may ask for the vader btl 
to be selected: mpirun  -mca pml obi1 -mca btl vader  …
Thanks,

_MAC

From: users [mailto:users-boun...@lists.open-mpi.org] On Behalf Of Jingchao 
Zhang
Sent: Wednesday, February 08, 2017 1:39 PM
To: Open MPI Users 
Subject: Re: [OMPI users] openmpi single node jobs using btl openib

Hi Gilles,

- btl/vader has a higher exclusivity than btl/sm, so bottom line, vader should 
be used instead of sm

Yes, I believe vader is being used in our case. I should say self,vader in my 
previous posts.

- is your interconnect infiniband or qlogic ? infiniband uses pml/ob1 and 
btl/openib for inter node communication, whereas qlogic users pml/cm and 
mtl/psm.
We have two fabrics in the cluster.  The older/original nodes are
interconnected on a Qlogic/Intel Truescale fabric.  We've expanded the cluster
with newer nodes that are interconnected with Intel Omni-path architecture
fabric.  The nodes can present to user-space the faster/preferred PSM/PSM2
and the IB/verbs interfaces.  We build ompi such that it has the freedom
to use the best transport, ie our build options: --with-verbs --with-psm
--with-psm2.  Perhaps we should stop building with verbs, but it is our
understanding that ompi should select the best transport at initialization.

- does your program involve MPI_Comm_spawn ? note that nor btl/vader nor btl/sm 
can be user for inter job communications (e.g. the main task and a spawn-ed 
task), so btl/openib would be used even for intra node communications.

From what we checked in cp2k source code, it doesn't have any 'MPI_Comm_spawn'. 
If we disable the use of openib within the single node run, and it performs 
much better/as expected, what is being used for inter job communication?  Is 
there a flag to specify that outputs rank-to-rank ompi transport selection?  
That would be very handy to have if it doesn't exist.

can you please run again your app with mpirun --mca pml_base_verbose 10 --mca 
btl_base_verbose 10 --mca mtl_base_verbose 10 ...

Please see attached for the verbose results.

Thanks,

Dr. Jingchao Zhang
Holland Computing Center
University of Nebraska-Lincoln
402-472-6400

From: users 
> on 
behalf of Gilles Gouaillardet >
Sent: Tuesday, February 7, 2017 6:37:21 PM
To: Open MPI Users
Subject: Re: [OMPI users] openmpi single node jobs using btl openib

Hi,

there are several uncommon things happening here :

- btl/vader has a higher exclusivity than btl/sm, so bottom line, vader should 
be used instead of sm

- is your interconnect infiniband or qlogic ? infiniband uses pml/ob1 and 
btl/openib for inter node communication,

  whereas qlogic users pml/cm and mtl/psm.

- does your program involve MPI_Comm_spawn ? note that nor btl/vader nor btl/sm 
can be user for inter job communications

(e.g. the main task and a spawn-ed task), so btl/openib would be used even for 
intra node communications.

can you please run again your app with

mpirun --mca pml_base_verbose 10 --mca btl_base_verbose 10 --mca 
mtl_base_verbose 10 ...

Cheers,

Gilles

On 2/8/2017 6:50 AM, Jingchao Zhang wrote:

Hi Jeff,

I just installed Open MPI: 2.0.2 (repo revision: v2.0.1-348-ge291d0e; release 
date: Jan 31, 2017) but have the same problem.

Attached please find two gdb backtraces on any write of a file descriptor 
returned from opening /dev/infiniband/uverbs in the cp2k.popt process.

Thanks,

Jingchao

From: users 
 on 
behalf of Jeff Squyres (jsquyres) 

Sent: Tuesday, February 7, 2017 2:14:40 PM
To: Open MPI User's List
Subject: Re: [OMPI users] openmpi single node jobs using btl openib

Can you try upgrading to Open MPI v2.0.2?  We just released that last week with 
a bunch of bug fixes.

> On Feb 7,

Re: [OMPI users] Severe performance issue with PSM2 and single-node CP2K jobs

2017-02-08 Thread Cabral, Matias A

Hi Hristo,

As you mention I have seen that the sm btl shows better performance for smaller 
messages than PMS2 shm device does, by running some osu benchmarks (especially 
BW for msg<256B). I even suspect that the difference would be more notable if 
you test the vader btl.  However, the piece that I would need to look at is 
what is CP2K particularly doing to make this difference so bad. 
As per today, unfortunately OMPI will not allow you to use the sm btl and the 
psm2 mtl simultaneously. Additionally, forcing to use the OmniPath verbs API 
(openib btl) will run much slower (this is not intended to be the API for 
message passing).

As mentioned above, I'll be looking at CP2k. 

Thanks, 

_MAC


-Original Message-
From: users [mailto:users-boun...@lists.open-mpi.org] On Behalf Of Iliev, Hristo
Sent: Wednesday, February 08, 2017 8:44 AM
To: users@lists.open-mpi.org
Subject: [OMPI users] Severe performance issue with PSM2 and single-node CP2K 
jobs

Hi,

While trying to debug a severe performance regression of CP2K runs with Open 
MPI 1.10.4 on our new cluster, after reproducing the problem with single-node 
jobs too, we found out that the root cause is that the presence of Intel 
Omni-Path hardware triggers the use of the cm PML and consequently the use of
psm2 MTL for shared-memory communication instead of the sm BTL. As subsequent 
tests with NetPIPE on a single socket showed (see the attached graph), the 
ping-pong latency of PSM2's shared-memory implementation is always 
significantly higher (20-60%) except for a relatively narrow range of message 
lengths 10-100 KiB, for which it is faster. Tests with processes on two sockets 
show that sm outperforms psm2 with smaller message sizes and psm2 outperforms 
sm for larger message sizes, at least for messages of less than 32 MiB. The 
real problem is though that the ScaLAPACK routines used by CP2K further 
exaggerate the difference, which results in orders of magnitude slower 
execution. We've tested it with both MKL and ScaLAPACK (and even BLAS) from 
Netlib in order to exclude possible performance regressions in MKL when used 
with Open MPI, which is our default configuration.

While disabling the psm2 MTL or enforcing the ob1 PML is a viable workaround 
for single-node jobs, it is not really a solution to our problem in general as 
utilising Omni-Path via its InfiniBand interface results in high latency and 
poor network bandwidth. As expected, disabling the "shm" device of PSM2 crashes 
the program.

My question is actually whether it is currently possible for several PMLs to 
coexist and to be used at the same time? Ideally, ob1 driving the sm BTL for 
intranode communication and cm driving the psm2 MTL for internode 
communication. From my limited understanding of the Open MPI source code, that 
doesn't really seem possible. While the psm2 MTL appears to be a relatively 
thin wrapper around the PSM2 API and therefore the problem might not really be 
in Open MPI but in the PSM2 library itself, it somehow does not affect Intel 
MPI. It seems to be a CP2K specific problem as a different software (Quantum 
ESPRESSO built with ScaLAPACK) runs fine, but then it could be the due to 
different ScaLAPACK routines being used.

The attached graphs show the ratio of the MPI ping-pong latency as measured by 
NetPIPE when run as follows:

mpiexec -n 2 --map-by core/socket --bind-to core NPmpi -a -I -l 1 -u 33554432

with and without --mca pml ob1. I also performed tests with Linux CMA support 
in PSM2 switched on and off (it is on by default), which doesn't change much. 
Our default Open MPI is built without CMA support.

Has anyone successfully run ScaLAPACK applications, and CP2K in particular, on 
systems with Intel Omni-Path? Perhaps I'm missing something here?

I'm sorry if this has already been discussed here. I went through the list 
archives, but couldn't find anything. If it was, I would be grateful if anyone 
could provide pointers to the relevant thread(s).

Kind regards,
Hristo
--
Hristo Iliev, PhD
JARA-HPC CSG "Parallel Efficiency"

IT Center
Group: High Performance Computing
Division: Computational Science and Engineering RWTH Aachen University 
Seffenter Weg 23
52074 Aachen, Germany
Tel: +49 (241) 80-24367
Fax: +49 (241) 80-624367
il...@itc.rwth-aachen.de
http://www.itc.rwth-aachen.de


___
users mailing list
users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

Re: [OMPI users] Open MPI Java Error

2017-02-08 Thread Cabral, Matias A

Hi Thyago,

psm is the user library to run with Intel TruScale cards.
psm2 is for Intel OmniPath.

There is a current problem in the libraries with OMPI java bindings:
https://www.open-mpi.org/faq/?category=java#java_limitations

thanks,

_MAC

From: users [mailto:users-boun...@lists.open-mpi.org] On Behalf Of Mota, Thyago
Sent: Wednesday, February 08, 2017 9:53 AM
To: Open MPI Users 
Subject: Re: [OMPI users] Open MPI Java Error

./configure --enable-mpi-java 
--with-jdk-dir=/Library/Java/JavaVirtualMachines/jdk1.8.0_121.jdk/Contents/Home/

What is psm?

On Wed, Feb 8, 2017 at 12:40 PM, Graham, Nathaniel Richard 
> wrote:

Hello Thyago,



What is your configure command?  Do you know if you are using psm or psm2?



-Nathan



--
Nathaniel Graham
HPC-DES
Los Alamos National Laboratory

From: users 
> on 
behalf of Mota, Thyago >
Sent: Wednesday, February 8, 2017 10:01 AM
To: users@lists.open-mpi.org
Subject: [OMPI users] Open MPI Java Error

I recently installed Open MPI and when I try to run mpirun I get the following 
error:


ORTE_ERROR_LOG: Bad parameter in file orted/pmix/pmix_server.c at line 262

ORTE_ERROR_LOG: Bad parameter in file ess_hnp_module.c at line 666

--

It looks like orte_init failed for some reason; your parallel process is

likely to abort.  There are many reasons that a parallel process can

fail during orte_init; some of which are due to configuration or

environment problems.  This failure appears to be an internal failure;

here's some additional information (which may only be relevant to an

Open MPI developer):

  pmix server init failed

  --> Returned value Bad parameter (-5) instead of ORTE_SUCCESS

--
Mota, Thyago
Moravian College

___
users mailing list
users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/users



--
Mota, Thyago
mo...@moravian.edu
Assistant Professor of Computer Science
Office: PPHAC 220
Phone: (610) 861-1403
Moravian College
___
users mailing list
users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

Re: [OMPI users] Error using hpcc benchmark

2017-02-01 Thread Cabral, Matias A

Hi Wodel,

As you already figured out, mpirun -x

Re: [OMPI users] Error using hpcc benchmark

2017-01-31 Thread Cabral, Matias A

Hi Wodel,

As Howard mentioned, this is probably because many ranks and sending to a 
single one and exhausting the receive requests MQ. You can individually enlarge 
the receive/send requests queues with the specific variables 
(PSM_MQ_RECVREQS_MAX/ PSM_MQ_SENDREQS_MAX) or increase both with 
PSM_MEMORY=max.  Note that the psm library will allocate more system memory for 
the queues.

Thanks,

_MAC

From: users [mailto:users-boun...@lists.open-mpi.org] On Behalf Of Howard 
Pritchard
Sent: Tuesday, January 31, 2017 6:38 AM
To: Open MPI Users 
Subject: Re: [OMPI users] Error using hpcc benchmark

Hi Wodel

Randomaccess part of HPCC is probably causing this.

Perhaps set PSM env. variable -

Export PSM_MQ_REVCREQ_MAX=1000

or something like that.

Alternatively launch the job using

mpirun --mca plm ob1 --host 

to avoid use of psm.  Performance will probably suffer with this option however.

Howard
wodel youchi > schrieb am 
Di. 31. Jan. 2017 um 08:27:
Hi,
I am n newbie in HPC world
I am trying to execute the hpcc benchmark on our cluster, but every time I 
start the job, I get this error, then the job exits
compute017.22840Exhausted 1048576 MQ irecv request descriptors, which usually 
indicates a user program error or insufficient request descriptors 
(PSM_MQ_RECVREQS_MAX=1048576)
compute024.22840Exhausted 1048576 MQ irecv request descriptors, which usually 
indicates a user program error or insufficient request descriptors 
(PSM_MQ_RECVREQS_MAX=1048576)
compute019.22847Exhausted 1048576 MQ irecv request descriptors, which usually 
indicates a user program error or insufficient request descriptors 
(PSM_MQ_RECVREQS_MAX=1048576)
---
Primary job  terminated normally, but 1 process returned
a non-zero exit code.. Per user-direction, the job has been aborted.
---
--
mpirun detected that one or more processes exited with non-zero status, thus 
causing
the job to be terminated. The first process to do so was:

  Process name: [[19601,1],272]
  Exit code:255
--
Platform : IBM PHPC
OS : RHEL 6.5
one management node
32 compute node : 16 cores, 32GB RAM, intel qlogic QLE7340 one port QRD 
infiniband 40Gb/s
I compiled hpcc against : IBM MPI, Openmpi 2.0.1 (compiled with gcc 4.4.7) and 
Openmpi 1.8.1 (compiled with gcc 4.4.7)
I get the errors, but each time on different compute nodes.
This is the command I used to start the job
mpirun -np 512 --mca mtl psm --hostfile hosts32 
/shared/build/hpcc-1.5.0b-blas-ompi-181/hpcc hpccinf.txt

Any help will be appreciated, and if you need more details, let me know.
Thanks in advance.

Regards.
___
users mailing list
users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/users
___
users mailing list
users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

Re: [OMPI users] device failed to appear .. Connection timed out

2016-12-08 Thread Cabral, Matias A

>Anyway,  /dev/hfi1_0 doesn't exist.
Make sure you have the hfi1 module/driver loaded.
In addition, please confirm the links are in active state on all the nodes 
`opainfo`

_MAC

From: users [mailto:users-boun...@lists.open-mpi.org] On Behalf Of Howard 
Pritchard
Sent: Thursday, December 08, 2016 9:23 AM
To: Open MPI Users 
Subject: Re: [OMPI users] device failed to appear .. Connection timed out

hello Daniele,

Could you post the output from ompi_info command?  I'm noticing on the RPMS 
that came with the rhel7.2 distro on
one of our systems that it was built to support psm2/hfi-1.

Two things, could you try running applications with

mpirun --mca pml ob1 (all the rest of your args)

and see if that works?

Second,  what sort of system are you using?  Is this a cluster?  If it is, you 
may want to check whether
you have a situation where its an omnipath interconnect and you have the 
psm2/hfi1 packages installed
but for some reason the omnipath HCAs themselves are not active.

On one of our omnipath systems the following hfi1 related pms are installed:

hfidiags-0.8-13.x86_64

hfi1-psm-devel-0.7-244.x86_64
libhfi1verbs-0.5-16.el7.x86_64
hfi1-psm-0.7-244.x86_64
hfi1-firmware-0.9-36.noarch
hfi1-psm-compat-0.7-244.x86_64
libhfi1verbs-devel-0.5-16.el7.x86_64
hfi1-0.11.3.10.0_327.el7.x86_64-245.x86_64
hfi1-firmware_debug-0.9-36.noarc
hfi1-diagtools-sw-0.8-13.x86_64

Howard

2016-12-08 8:45 GMT-07:00 r...@open-mpi.org 
>:
Sounds like something didn’t quite get configured right, or maybe you have a 
library installed that isn’t quite setup correctly, or...

Regardless, we generally advise building from source to avoid such problems. Is 
there some reason not to just do so?

On Dec 8, 2016, at 6:16 AM, Daniele Tartarini 
> wrote:

Hi,

I've installed on a Red Hat 7.2 the OpenMPI distributed via Yum:

openmpi-devel.x86_64 1.10.3-3.el7

any code I try to run (including the mpitests-*) I get the following message 
with slight variants:

 my_machine.171619hfi_wait_for_device: The /dev/hfi1_0 device failed to 
appear after 15.0 seconds: Connection timed out

Is anyone able to help me in identifying the source of the problem?
Anyway,  /dev/hfi1_0 doesn't exist.

If I use an OpenMPI version compiled from source I have no issue (gcc 4.8.5).

many thanks in advance.

cheers
Daniele
___
users mailing list
users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

___
users mailing list
users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

___
users mailing list
users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

Re: [OMPI users] Openmpi 2.0.1 build --with-psm2 failed on CentOS 7.2

2016-10-12 Thread Cabral, Matias A

Hi Limin,

One more detail. I advise to use a stable release:

https://github.com/01org/opa-psm2/releases

Regards,

_MAC

From: users [mailto:users-boun...@lists.open-mpi.org] On Behalf Of Limin Gu
Sent: Tuesday, October 11, 2016 7:33 PM
To: Open MPI Users <users@lists.open-mpi.org>
Subject: Re: [OMPI users] Openmpi 2.0.1 build --with-psm2 failed on CentOS 7.2

Thank you very much, MAC!

Limin




On Tue, Oct 11, 2016 at 10:15 PM, Cabral, Matias A 
<matias.a.cab...@intel.com<mailto:matias.a.cab...@intel.com>> wrote:
Building psm2 should not be complicated (in case you cannot find a newer 
binary):

https://github.com/01org/opa-psm2


Note that newer rpm are named hfi1-psm*


_MAC

From: users 
[mailto:users-boun...@lists.open-mpi.org<mailto:users-boun...@lists.open-mpi.org>]
 On Behalf Of Limin Gu
Sent: Tuesday, October 11, 2016 6:44 PM
To: Open MPI Users <users@lists.open-mpi.org<mailto:users@lists.open-mpi.org>>
Subject: Re: [OMPI users] Openmpi 2.0.1 build --with-psm2 failed on CentOS 7.2


Thanks Gilles!



Limin

On Tue, Oct 11, 2016 at 9:33 PM, Gilles Gouaillardet 
<gil...@rist.or.jp<mailto:gil...@rist.or.jp>> wrote:

Limin,



It seems libpsm2 provided by Centos 7 is a bit too old

all symbols are prefixed with psm_, and Open MPI expect they are prefixed with 
psm2_

i am afraid your only option is to manually install the latest libpsm2 and then 
configure again with your psm2 install dir


Cheers,

Gilles

On 10/12/2016 9:57 AM, Limin Gu wrote:
Hi MAC,

It seems /usr/lib64/libpsm2.so.2 has no symbols. Can configure check some other 
ways?



[root@uranus ~]# rpm -qi libpsm2-0.7-4.el7.x86_64

Name: libpsm2

Version : 0.7

Release : 4.el7

Architecture: x86_64

Install Date: Tue 11 Oct 2016 05:45:59 PM PDT

Group   : System Environment/Libraries

Size: 400282

License : GPLv2 or BSD

Signature   : RSA/SHA256, Wed 25 Nov 2015 07:02:20 AM PST, Key ID 
24c6a8a7f4a80eb5

Source RPM  : libpsm2-0.7-4.el7.src.rpm

Build Date  : Fri 20 Nov 2015 08:05:13 AM PST

Build Host  : worker1.bsys.centos.org<http://worker1.bsys.centos.org>

Relocations : (not relocatable)

Packager: CentOS BuildSystem <http://bugs.centos.org>

Vendor  : CentOS

URL : http://www.intel.com/

Summary : Intel PSM Libraries

Description :

The PSM Messaging API, or PSM API, is Intel's low-level

user-level communications interface for the Truescale

family of products. PSM users are enabled with mechanisms

necessary to implement higher level communications

interfaces in parallel environments.

[root@uranus ~]# objdump -p /usr/lib64/libpsm2.so.2 |grep SONAME

  SONAME   libpsm2.so.2

[root@uranus ~]# nm /usr/lib64/libpsm2.so.2

nm: /usr/lib64/libpsm2.so.2: no symbols

[root@uranus ~]#


Thanks!
Limin


On Tue, Oct 11, 2016 at 7:00 PM, Cabral, Matias A 
<matias.a.cab...@intel.com<mailto:matias.a.cab...@intel.com>> wrote:
Hi  Limin,

psm2_mq_irecv2 should be in libpsm2.so.  I’m not quite sure how CentOS packs it 
so I would like a little more info about the version being used. Some things to 
share:

>rpm -qi libpsm2-0.7-4.el7.x86_64
> objdump –p /usr/lib64/libpsm2.so |grep SONAME
>nm /usr/lib64/libpsm2.so |grep psm2_mq_irecv2 (will not work if the lib 
>Stripped)


Thanks,
_MAC

From: users 
[mailto:users-boun...@lists.open-mpi.org<mailto:users-boun...@lists.open-mpi.org>]
 On Behalf Of Limin Gu
Sent: Tuesday, October 11, 2016 2:58 PM
To: Open MPI Users <users@lists.open-mpi.org<mailto:users@lists.open-mpi.org>>
Subject: [OMPI users] Openmpi 2.0.1 build --with-psm2 failed on CentOS 7.2

Hi All,

I am trying to build openmpi 2.0.1 on a CentOS 7.2 system, and I have following 
libpsm2 packages installed:

libpsm2-0.7-4.el7.x86_64
libpsm2-compat-0.7-4.el7.x86_64
libpsm2-compat-devel-0.7-4.el7.x86_64
libpsm2-devel-0.7-4.el7.x86_64

I added --with-psm2 to my configure, but it failed:

--- MCA component mtl:psm2 (m4 configuration macro)
checking for MCA component mtl:psm2 compile mode... static
checking --with-psm2 value... simple ok (unspecified)
checking --with-psm2-libdir value... simple ok (unspecified)
checking psm2.h usability... yes
checking psm2.h presence... yes
checking for psm2.h... yes
looking for library without search path
checking for library containing psm2_mq_irecv2... no
configure: error: PSM2 support requested but not found.  Aborting
error: Bad exit status from /var/tmp/rpm-tmp.TLxu8O (%build)


/usr/lib64/libpsm2.so is on the system though.

What else libraries do I need for psm2?

Thank you!


___
users mailing list
users@lists.open-mpi.org<mailto:users@lists.open-mpi.org>
https://rfd.newmexicoconsortium.org/mailman/listinfo/users



___

users mailing list

users@lists.open-mpi.org<mailto:users@lists.open-mpi.org>

https://rfd.newmexicoconsortium.org/ma

Re: [OMPI users] Openmpi 2.0.1 build --with-psm2 failed on CentOS 7.2

2016-10-11 Thread Cabral, Matias A

Building psm2 should not be complicated (in case you cannot find a newer 
binary):

https://github.com/01org/opa-psm2


Note that newer rpm are named hfi1-psm*


_MAC

From: users [mailto:users-boun...@lists.open-mpi.org] On Behalf Of Limin Gu
Sent: Tuesday, October 11, 2016 6:44 PM
To: Open MPI Users <users@lists.open-mpi.org>
Subject: Re: [OMPI users] Openmpi 2.0.1 build --with-psm2 failed on CentOS 7.2


Thanks Gilles!



Limin

On Tue, Oct 11, 2016 at 9:33 PM, Gilles Gouaillardet 
<gil...@rist.or.jp<mailto:gil...@rist.or.jp>> wrote:

Limin,



It seems libpsm2 provided by Centos 7 is a bit too old

all symbols are prefixed with psm_, and Open MPI expect they are prefixed with 
psm2_

i am afraid your only option is to manually install the latest libpsm2 and then 
configure again with your psm2 install dir


Cheers,

Gilles

On 10/12/2016 9:57 AM, Limin Gu wrote:
Hi MAC,

It seems /usr/lib64/libpsm2.so.2 has no symbols. Can configure check some other 
ways?



[root@uranus ~]# rpm -qi libpsm2-0.7-4.el7.x86_64

Name: libpsm2

Version : 0.7

Release : 4.el7

Architecture: x86_64

Install Date: Tue 11 Oct 2016 05:45:59 PM PDT

Group   : System Environment/Libraries

Size: 400282

License : GPLv2 or BSD

Signature   : RSA/SHA256, Wed 25 Nov 2015 07:02:20 AM PST, Key ID 
24c6a8a7f4a80eb5

Source RPM  : libpsm2-0.7-4.el7.src.rpm

Build Date  : Fri 20 Nov 2015 08:05:13 AM PST

Build Host  : worker1.bsys.centos.org<http://worker1.bsys.centos.org>

Relocations : (not relocatable)

Packager: CentOS BuildSystem <http://bugs.centos.org>

Vendor  : CentOS

URL : http://www.intel.com/

Summary : Intel PSM Libraries

Description :

The PSM Messaging API, or PSM API, is Intel's low-level

user-level communications interface for the Truescale

family of products. PSM users are enabled with mechanisms

necessary to implement higher level communications

interfaces in parallel environments.

[root@uranus ~]# objdump -p /usr/lib64/libpsm2.so.2 |grep SONAME

  SONAME   libpsm2.so.2

[root@uranus ~]# nm /usr/lib64/libpsm2.so.2

nm: /usr/lib64/libpsm2.so.2: no symbols

[root@uranus ~]#


Thanks!
Limin


On Tue, Oct 11, 2016 at 7:00 PM, Cabral, Matias A 
<matias.a.cab...@intel.com<mailto:matias.a.cab...@intel.com>> wrote:
Hi  Limin,

psm2_mq_irecv2 should be in libpsm2.so.  I’m not quite sure how CentOS packs it 
so I would like a little more info about the version being used. Some things to 
share:

>rpm -qi libpsm2-0.7-4.el7.x86_64
> objdump –p /usr/lib64/libpsm2.so |grep SONAME
>nm /usr/lib64/libpsm2.so |grep psm2_mq_irecv2 (will not work if the lib 
>Stripped)


Thanks,
_MAC

From: users 
[mailto:users-boun...@lists.open-mpi.org<mailto:users-boun...@lists.open-mpi.org>]
 On Behalf Of Limin Gu
Sent: Tuesday, October 11, 2016 2:58 PM
To: Open MPI Users <users@lists.open-mpi.org<mailto:users@lists.open-mpi.org>>
Subject: [OMPI users] Openmpi 2.0.1 build --with-psm2 failed on CentOS 7.2

Hi All,

I am trying to build openmpi 2.0.1 on a CentOS 7.2 system, and I have following 
libpsm2 packages installed:

libpsm2-0.7-4.el7.x86_64
libpsm2-compat-0.7-4.el7.x86_64
libpsm2-compat-devel-0.7-4.el7.x86_64
libpsm2-devel-0.7-4.el7.x86_64

I added --with-psm2 to my configure, but it failed:

--- MCA component mtl:psm2 (m4 configuration macro)
checking for MCA component mtl:psm2 compile mode... static
checking --with-psm2 value... simple ok (unspecified)
checking --with-psm2-libdir value... simple ok (unspecified)
checking psm2.h usability... yes
checking psm2.h presence... yes
checking for psm2.h... yes
looking for library without search path
checking for library containing psm2_mq_irecv2... no
configure: error: PSM2 support requested but not found.  Aborting
error: Bad exit status from /var/tmp/rpm-tmp.TLxu8O (%build)


/usr/lib64/libpsm2.so is on the system though.

What else libraries do I need for psm2?

Thank you!


___
users mailing list
users@lists.open-mpi.org<mailto:users@lists.open-mpi.org>
https://rfd.newmexicoconsortium.org/mailman/listinfo/users



___

users mailing list

users@lists.open-mpi.org<mailto:users@lists.open-mpi.org>

https://rfd.newmexicoconsortium.org/mailman/listinfo/users


___
users mailing list
users@lists.open-mpi.org<mailto:users@lists.open-mpi.org>
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

___
users mailing list
users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

Re: [OMPI users] Openmpi 2.0.1 build --with-psm2 failed on CentOS 7.2

2016-10-11 Thread Cabral, Matias A

Hi  Limin,

psm2_mq_irecv2 should be in libpsm2.so.  I’m not quite sure how CentOS packs it 
so I would like a little more info about the version being used. Some things to 
share:

>rpm -qi libpsm2-0.7-4.el7.x86_64
> objdump –p /usr/lib64/libpsm2.so |grep SONAME
>nm /usr/lib64/libpsm2.so |grep psm2_mq_irecv2 (will not work if the lib 
>Stripped)


Thanks,
_MAC

From: users [mailto:users-boun...@lists.open-mpi.org] On Behalf Of Limin Gu
Sent: Tuesday, October 11, 2016 2:58 PM
To: Open MPI Users 
Subject: [OMPI users] Openmpi 2.0.1 build --with-psm2 failed on CentOS 7.2

Hi All,

I am trying to build openmpi 2.0.1 on a CentOS 7.2 system, and I have following 
libpsm2 packages installed:

libpsm2-0.7-4.el7.x86_64
libpsm2-compat-0.7-4.el7.x86_64
libpsm2-compat-devel-0.7-4.el7.x86_64
libpsm2-devel-0.7-4.el7.x86_64

I added --with-psm2 to my configure, but it failed:

--- MCA component mtl:psm2 (m4 configuration macro)
checking for MCA component mtl:psm2 compile mode... static
checking --with-psm2 value... simple ok (unspecified)
checking --with-psm2-libdir value... simple ok (unspecified)
checking psm2.h usability... yes
checking psm2.h presence... yes
checking for psm2.h... yes
looking for library without search path
checking for library containing psm2_mq_irecv2... no
configure: error: PSM2 support requested but not found.  Aborting
error: Bad exit status from /var/tmp/rpm-tmp.TLxu8O (%build)


/usr/lib64/libpsm2.so is on the system though.

What else libraries do I need for psm2?

Thank you!

___
users mailing list
users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

Re: [OMPI users] MPI_Comm_spawn

2016-09-29 Thread Cabral, Matias A

Hi Giles et.al.,

You are right, ptl.c is in PSM2 code. As Ralph mentions, dynamic process 
support was/is not working in OMPI when using PSM2 because of an issue related 
to the transport keys. This was fixed in PR #1602 
(https://github.com/open-mpi/ompi/pull/1602) and should be included in v2.0.2. 
HOWEVER, this not the error Juraj is seeing. The root of the assertion is 
because the PSM/PSM2 MTLs will check for where the “original” process are 
running and, if detects all are local to the node, it will ONLY initialize the 
shared memory device (variable PSM2_DEVICES="self,shm” ). This is to avoid 
“reserving” HW resources in the HFI card that wouldn’t be used unless you later 
on spawn ranks in other nodes.  Therefore, to allow dynamic process to be 
spawned on other nodes you need to tell PSM2 to instruct the HW to initialize 
all the de devices by making the environment variable 
PSM2_DEVICES="self,shm,hfi" available before running the job.
Note that setting PSM2_DEVICES (*) will solve the below assertion, you will 
most likely still see the transport key issue if PR1602 if is not included.

Thanks,

_MAC

(*)
PSM2_DEVICES  -> Omni Path
PSM_DEVICES  -> TrueScale

From: users [mailto:users-boun...@lists.open-mpi.org] On Behalf Of 
r...@open-mpi.org
Sent: Thursday, September 29, 2016 7:12 AM
To: Open MPI Users 
Subject: Re: [OMPI users] MPI_Comm_spawn

Ah, that may be why it wouldn’t show up in the OMPI code base itself. If that 
is the case here, then no - OMPI v2.0.1 does not support comm_spawn for PSM. It 
is fixed in the upcoming 2.0.2

On Sep 29, 2016, at 6:58 AM, Gilles Gouaillardet 
> wrote:

Ralph,

My guess is that ptl.c comes from PSM lib ...

Cheers,

Gilles

On Thursday, September 29, 2016, r...@open-mpi.org 
> wrote:
Spawn definitely does not work with srun. I don’t recognize the name of the 
file that segfaulted - what is “ptl.c”? Is that in your manager program?

On Sep 29, 2016, at 6:06 AM, Gilles Gouaillardet 
>
 wrote:

Hi,

I do not expect spawn can work with direct launch (e.g. srun)

Do you have PSM (e.g. Infinipath) hardware ? That could be linked to the failure

Can you please try

mpirun --mca pml ob1 --mca btl tcp,sm,self -np 1 --hostfile my_hosts ./manager 1

and see if it help ?

Note if you have the possibility, I suggest you first try that without slurm, 
and then within a slurm job

Cheers,

Gilles

On Thursday, September 29, 2016, 
juraj2...@gmail.com 
> 
wrote:
Hello,

I am using MPI_Comm_spawn to dynamically create new processes from single 
manager process. Everything works fine when all the processes are running on 
the same node. But imposing restriction to run only a single process per node 
does not work. Below are the errors produced during multinode interactive 
session and multinode sbatch job.

The system I am using is: Linux version 3.10.0-229.el7.x86_64 
(buil...@kbuilder.dev.centos.org) (gcc 
version 4.8.2 20140120 (Red Hat 4.8.2-16) (GCC) )
I am using Open MPI 2.0.1
Slurm is version 15.08.9

What is preventing my jobs to spawn on multiple nodes? Does slurm requires some 
additional configuration to allow it? Is it issue on the MPI side, does it need 
to be compiled with some special flag (I have compiled it with 
--enable-mpi-fortran=all --with-pmi)?

The code I am launching is here: https://github.com/goghino/dynamicMPI

Manager tries to launch one new process (./manager 1), the error produced by 
requesting each process to be located on different node (interactive session):
$ salloc -N 2
$ cat my_hosts
icsnode37
icsnode38
$ mpirun -np 1 -npernode 1 --hostfile my_hosts ./manager 1
[manager]I'm running MPI 3.1
[manager]Runing on node icsnode37
icsnode37.12614Assertion failure at ptl.c:183: epaddr == ((void *)0)
icsnode38.32443Assertion failure at ptl.c:183: epaddr == ((void *)0)
[icsnode37:12614] *** Process received signal ***
[icsnode37:12614] Signal: Aborted (6)
[icsnode37:12614] Signal code:  (-6)
[icsnode38:32443] *** Process received signal ***
[icsnode38:32443] Signal: Aborted (6)
[icsnode38:32443] Signal code:  (-6)

The same example as above via sbatch job submission:
$ cat job.sbatch
#!/bin/bash

#SBATCH --nodes=2
#SBATCH --ntasks-per-node=1

module load openmpi/2.0.1
srun -n 1 -N 1 ./manager 1

$ cat output.o
[manager]I'm running MPI 3.1
[manager]Runing on node icsnode39
srun: Job step aborted: Waiting up to 32 seconds for job step to finish.
[icsnode39:9692] *** An error occurred in MPI_Comm_spawn
[icsnode39:9692] *** reported by process [1007812608,0]
[icsnode39:9692] *** on communicator MPI_COMM_SELF

Re: [OMPI users] runtime performance tuning for Intel OMA interconnect

2016-08-11 Thread Cabral, Matias A

Michael,

In general terms and assuming you are running all messages sizes in PIO Eager 
Mode, the communication is going to be affected by the CPU load. In other 
words, the bigger the message, the more CPU cycles to copy the buffer. 
Additionally, I have to say I’m not very certain how MPI_Send() will behave 
under the hood with temporary buffering. I think a more predicable behavior 
would be seen with MPI_Ssend().  Now, if you really don’t want to see the 
sender affected by the receiver load, you need to move to non-blocking calls 
MPI_Isend().


_MAC

From: users [mailto:users-boun...@lists.open-mpi.org] On Behalf Of Xiaolong Cui
Sent: Thursday, August 11, 2016 2:13 PM
To: Open MPI Users <users@lists.open-mpi.org>
Subject: Re: [OMPI users] runtime performance tuning for Intel OMA interconnect

Sorry, forgot the attachments.

On Thu, Aug 11, 2016 at 5:06 PM, Xiaolong Cui 
<sunshine...@gmail.com<mailto:sunshine...@gmail.com>> wrote:
Thanks! I tried it, but it didn't solve my problem. Maybe the reason is not 
eager/rndv.

The reason why I want to always use eager mode is that I want to avoid a sender 
being slowed down by an unready receiver. I can prevent a sender from slowing 
down by always using eager mode on InfiniBand, just like your approach, but I 
cannot repeat this on OPA. Based on the experiments below, it seems to me that 
a sender will be delayed to some extent due to reasons other than eager/rndv.

I designed a simple test (see hello_world.c in attachment) where there is one 
sender rank (r0) and one receiver rank (r1). r0 always runs at full speed, but 
r1 runs at full speed in one case and half speed in the second case. To run r1 
at half speed, I collate a third rank r2 with r1 (see rankfile). Then I compare 
the completion time at r0 to see if there is a slow down when r1 is "unready to 
receive". The result is positive. But it is surprising that the delay varies 
significantly when I change the message length. This is different from my 
previous observation when eager/rndv is the cause.

So my question is do you know other factors that cause a delay to a MPI_Send() 
when the receiver is not ready to receive?




On Wed, Aug 10, 2016 at 11:48 PM, Cabral, Matias A 
<matias.a.cab...@intel.com<mailto:matias.a.cab...@intel.com>> wrote:
To remain in eager mode you need to increase the size of 
PSM2_MQ_RNDV_HFI_THRESH.
PSM2_MQ_EAGER_SDMA_SZ is the threshold at which PSM changes from PIO (uses the 
CPU) to start setting SDMA engines.  This summary may help:

PIO Eager Mode:  0 bytes -> PSM2_MQ_EAGER_SDMA_SZ - 1
SDMA Eager Mode:PSM2_MQ_EAGER_SDMA_SZ -> PSM2_MQ_RNDV_HFI_THRESH - 1
RNDZ Expected:   PSM2_MQ_RNDV_HFI_THRESH -> Largest supported value.

Regards,

_MAC

From: users 
[mailto:users-boun...@lists.open-mpi.org<mailto:users-boun...@lists.open-mpi.org>]
 On Behalf Of Xiaolong Cui
Sent: Wednesday, August 10, 2016 7:19 PM
To: Open MPI Users <users@lists.open-mpi.org<mailto:users@lists.open-mpi.org>>
Subject: Re: [OMPI users] runtime performance tuning for Intel OMA interconnect

Hi Matias,

Thanks a lot, that's very helpful!

What I need indeed is to always use eager mode. But I didn't find any 
information about PSM2_MQ_EAGER_SDMA_SZ online. Would you please elaborate on 
"Just in case PSM2_MQ_EAGER_SDMA_SZ changes PIO to SDMA, always in eager mode."

Thanks!
Michael

On Wed, Aug 10, 2016 at 3:59 PM, Cabral, Matias A 
<matias.a.cab...@intel.com<mailto:matias.a.cab...@intel.com>> wrote:
Hi Michael,

When Open MPI run on Omni-Path it will choose the PSM2 MTL by default, to use 
the libpsm2.so. Strictly speaking, it has compatibility to run using the openib 
BTL. However, the performance so significantly impacted that it is, not only 
discouraged, but no tuning would make sense. Regarding the PSM2 MTL, currently 
it only supports two mca parameters ("mtl_psm2_connect_timeout" and 
"mtl_psm2_priority") which are not for what you are looking for. Instead, you 
can set values directly in the PSM2 library with environment variables.  
Further info in the Programmers Guide:

http://www.intel.com/content/dam/support/us/en/documents/network-and-i-o/fabric-products/Intel_PSM2_PG_H76473_v3_0.pdf
More docs:

https://www-ssl.intel.com/content/www/us/en/support/network-and-i-o/fabric-products/16242.html?wapkw=psm2

Now, for your parameters:

btl = openib,vader,self  -> Ignore this one
btl_openib_eager_limit = 16   -> I don’t clearly see the diff with the 
below parameter. However, they are set to the same value. Just in case 
PSM2_MQ_EAGER_SDMA_SZ changes PIO to SDMA, always in eager mode.
btl_openib_rndv_eager_limit = 16  -> PSM2_MQ_RNDV_HFI_THRESH
btl_openib_max_send_size = 16   -> does not apply to PSM2
btl_openib_receive_queues = 
P,128,256,192,128:S,2048,1024,1008,64:S,12288,1024,1008,64:S,16,1024,512,512
  -> does not appl

Re: [OMPI users] runtime performance tuning for Intel OMA interconnect

2016-08-10 Thread Cabral, Matias A

To remain in eager mode you need to increase the size of 
PSM2_MQ_RNDV_HFI_THRESH.
PSM2_MQ_EAGER_SDMA_SZ is the threshold at which PSM changes from PIO (uses the 
CPU) to start setting SDMA engines.  This summary may help:

PIO Eager Mode:  0 bytes -> PSM2_MQ_EAGER_SDMA_SZ - 1
SDMA Eager Mode:PSM2_MQ_EAGER_SDMA_SZ -> PSM2_MQ_RNDV_HFI_THRESH - 1
RNDZ Expected:   PSM2_MQ_RNDV_HFI_THRESH -> Largest supported value.

Regards,

_MAC

From: users [mailto:users-boun...@lists.open-mpi.org] On Behalf Of Xiaolong Cui
Sent: Wednesday, August 10, 2016 7:19 PM
To: Open MPI Users <users@lists.open-mpi.org>
Subject: Re: [OMPI users] runtime performance tuning for Intel OMA interconnect

Hi Matias,

Thanks a lot, that's very helpful!

What I need indeed is to always use eager mode. But I didn't find any 
information about PSM2_MQ_EAGER_SDMA_SZ online. Would you please elaborate on 
"Just in case PSM2_MQ_EAGER_SDMA_SZ changes PIO to SDMA, always in eager mode."

Thanks!
Michael

On Wed, Aug 10, 2016 at 3:59 PM, Cabral, Matias A 
<matias.a.cab...@intel.com<mailto:matias.a.cab...@intel.com>> wrote:
Hi Michael,

When Open MPI run on Omni-Path it will choose the PSM2 MTL by default, to use 
the libpsm2.so. Strictly speaking, it has compatibility to run using the openib 
BTL. However, the performance so significantly impacted that it is, not only 
discouraged, but no tuning would make sense. Regarding the PSM2 MTL, currently 
it only supports two mca parameters ("mtl_psm2_connect_timeout" and 
"mtl_psm2_priority") which are not for what you are looking for. Instead, you 
can set values directly in the PSM2 library with environment variables.  
Further info in the Programmers Guide:

http://www.intel.com/content/dam/support/us/en/documents/network-and-i-o/fabric-products/Intel_PSM2_PG_H76473_v3_0.pdf
More docs:

https://www-ssl.intel.com/content/www/us/en/support/network-and-i-o/fabric-products/16242.html?wapkw=psm2

Now, for your parameters:

btl = openib,vader,self  -> Ignore this one
btl_openib_eager_limit = 16   -> I don’t clearly see the diff with the 
below parameter. However, they are set to the same value. Just in case 
PSM2_MQ_EAGER_SDMA_SZ changes PIO to SDMA, always in eager mode.
btl_openib_rndv_eager_limit = 16  -> PSM2_MQ_RNDV_HFI_THRESH
btl_openib_max_send_size = 16   -> does not apply to PSM2
btl_openib_receive_queues = 
P,128,256,192,128:S,2048,1024,1008,64:S,12288,1024,1008,64:S,16,1024,512,512
  -> does not apply for PSM2.

Thanks,
Regards,

_MAC
BTW, should change the subject OMA -> OPA


From: users [mailto:users-boun...@lists.open-mpi.org] On Behalf Of Xiaolong Cui
Sent: Tuesday, August 09, 2016 2:22 PM
To: users@lists.open-mpi.org<mailto:users@lists.open-mpi.org>
Subject: [OMPI users] runtime performance tuning for Intel OMA interconnect

I used to tune the performance of OpenMPI on InfiniBand by changing the OpenMPI 
MCA parameters for openib component (see 
https://www.open-mpi.org/faq/?category=openfabrics). Now I migrate to a new 
cluster that deploys Intel's omni-path interconnect, and my previous approach 
does not work any more. Does anyone know how to tune the performance for 
omni-path interconnect (what OpenMPI component to change) ?

The version of OpenMPI is openmpi-1.10.2-hfi. I have included the output from 
opmi_info and openib parameters that I used to change. Thanks!

Sincerely,
Michael

___
users mailing list
users@lists.open-mpi.org<mailto:users@lists.open-mpi.org>
https://rfd.newmexicoconsortium.org/mailman/listinfo/users




___
users mailing list
users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

Re: [OMPI users] runtime performance tuning for Intel OMA interconnect

2016-08-10 Thread Cabral, Matias A

Hi Michael,

When Open MPI run on Omni-Path it will choose the PSM2 MTL by default, to use 
the libpsm2.so. Strictly speaking, it has compatibility to run using the openib 
BTL. However, the performance so significantly impacted that it is, not only 
discouraged, but no tuning would make sense. Regarding the PSM2 MTL, currently 
it only supports two mca parameters ("mtl_psm2_connect_timeout" and 
"mtl_psm2_priority") which are not for what you are looking for. Instead, you 
can set values directly in the PSM2 library with environment variables.  
Further info in the Programmers Guide:

http://www.intel.com/content/dam/support/us/en/documents/network-and-i-o/fabric-products/Intel_PSM2_PG_H76473_v3_0.pdf
More docs:

https://www-ssl.intel.com/content/www/us/en/support/network-and-i-o/fabric-products/16242.html?wapkw=psm2

Now, for your parameters:

btl = openib,vader,self  -> Ignore this one
btl_openib_eager_limit = 16   -> I don’t clearly see the diff with the 
below parameter. However, they are set to the same value. Just in case 
PSM2_MQ_EAGER_SDMA_SZ changes PIO to SDMA, always in eager mode.
btl_openib_rndv_eager_limit = 16  -> PSM2_MQ_RNDV_HFI_THRESH
btl_openib_max_send_size = 16   -> does not apply to PSM2
btl_openib_receive_queues = 
P,128,256,192,128:S,2048,1024,1008,64:S,12288,1024,1008,64:S,16,1024,512,512
  -> does not apply for PSM2.

Thanks,
Regards,

_MAC
BTW, should change the subject OMA -> OPA


From: users [mailto:users-boun...@lists.open-mpi.org] On Behalf Of Xiaolong Cui
Sent: Tuesday, August 09, 2016 2:22 PM
To: users@lists.open-mpi.org
Subject: [OMPI users] runtime performance tuning for Intel OMA interconnect

I used to tune the performance of OpenMPI on InfiniBand by changing the OpenMPI 
MCA parameters for openib component (see 
https://www.open-mpi.org/faq/?category=openfabrics). Now I migrate to a new 
cluster that deploys Intel's omni-path interconnect, and my previous approach 
does not work any more. Does anyone know how to tune the performance for 
omni-path interconnect (what OpenMPI component to change) ?

The version of OpenMPI is openmpi-1.10.2-hfi. I have included the output from 
opmi_info and openib parameters that I used to change. Thanks!

Sincerely,
Michael
___
users mailing list
users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

Re: [OMPI users] PSM vs PSM2

2016-06-02 Thread Cabral, Matias A

Hi Durga,

Here is a short summary:
PSM:  is intended for Intel TrueScale InfiniBand product series. It is also 
known as PSM gen 1, uses libpsm_infinipath.so
PSM2: is intended for Intel’s next generation fabric called OmniPath. PSM gen2, 
uses libpsm2.so. I didn’t know about the owner.txt missing. I’ll look into that.

Thanks!

_MAC

From: users [mailto:users-boun...@open-mpi.org] On Behalf Of dpchoudh .
Sent: Thursday, June 02, 2016 3:57 PM
To: Open MPI Users 
Subject: [OMPI users] PSM vs PSM2

Hello all
What is the difference between PSM and PSM2? Any pointer to more information is 
appreciated. Also, the PSM2 MTL does not seem to have a owner.txt file (on 
master, at least). Why is that?
Thanks
Durga

We learn from history that we never learn from history.

Re: [OMPI users] locked memory and queue pairs

2016-03-17 Thread Cabral, Matias A

I was looking for lines like" [nodexyz:17085] selected cm best priority 40" and 
 " [nodexyz:17099] select: component psm selected"

_MAC


-Original Message-
From: users [mailto:users-boun...@open-mpi.org] On Behalf Of Michael Di Domenico
Sent: Thursday, March 17, 2016 5:52 AM
To: Open MPI Users <us...@open-mpi.org>
Subject: Re: [OMPI users] locked memory and queue pairs

On Wed, Mar 16, 2016 at 4:49 PM, Cabral, Matias A <matias.a.cab...@intel.com> 
wrote:
> I didn't go into the code to see who is actually calling this error message, 
> but I suspect this may be a generic error for "out of memory" kind of thing 
> and not specific to the que pair. To confirm please add  -mca 
> pml_base_verbose 100 and add  -mca mtl_base_verbose 100  to see what is being 
> selected.

this didn't spit out anything overly useful, just lots of lines

[node001:00909] mca: base: components_register: registering pml components 
[node001:00909] mca: base: components_register: found loaded component v 
[node001:00909] mca: base: components_register: component v register function 
successful [node001:00909] mca: base: components_register: found loaded 
component bfo [node001:00909] mca: base: components_register: component bfo 
register function successful [node001:00909] mca: base: components_register: 
found loaded component cm [node001:00909] mca: base: components_register: 
component cm register function successful [node001:00909] mca: base: 
components_register: found loaded component ob1 [node001:00909] mca: base: 
components_register: component ob1 register function successful

> I'm trying to remember some details of IMB  and alltoallv to see if it is 
> indeed requiring more resources that the other micro benchmarks.

i'm using IMB for my tests, but this issue came up because a researcher isn't 
able to run large alltoall codes, so i don't believe it's specific to IMB

> BTW, did you confirm the limits setup? Also do the nodes have all the same 
> amount of mem?

yes, all nodes have the limits set to unlimited and each node has 256GB of 
memory ___
users mailing list
us...@open-mpi.org
Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
Link to this post: 
http://www.open-mpi.org/community/lists/users/2016/03/28726.php

Re: [OMPI users] locked memory and queue pairs

2016-03-16 Thread Cabral, Matias A

I didn't go into the code to see who is actually calling this error message, 
but I suspect this may be a generic error for "out of memory" kind of thing and 
not specific to the que pair. To confirm please add  -mca pml_base_verbose 100 
and add  -mca mtl_base_verbose 100  to see what is being selected. 

I'm trying to remember some details of IMB  and alltoallv to see if it is 
indeed requiring more resources that the other micro benchmarks. 

BTW, did you confirm the limits setup? Also do the nodes have all the same 
amount of mem? 

_MAC

-Original Message-
From: users [mailto:users-boun...@open-mpi.org] On Behalf Of Michael Di Domenico
Sent: Wednesday, March 16, 2016 1:25 PM
To: Open MPI Users <us...@open-mpi.org>
Subject: Re: [OMPI users] locked memory and queue pairs

On Wed, Mar 16, 2016 at 3:37 PM, Cabral, Matias A <matias.a.cab...@intel.com> 
wrote:
> Hi Michael,
>
> I may be missing some context, if you are using the qlogic cards you will 
> always want to use the psm mtl (-mca pml cm -mca mtl psm) and not openib btl. 
> As Tom suggest, confirm the limits are setup on every node: could it be the 
> alltoall is reaching a node that "others" are not? Please share the command 
> line and the error message.

Yes, under normal circumstances, I use PSM.  i only disabled to see if it 
affected any kind of change.

the test i'm running is

mpirun -n 512 ./IMB-MPI1 alltoallv

when the system gets to 128 ranks, it freezes and errors out with

---

A process failed to create a queue pair. This usually means either the device 
has run out of queue pairs (too many connections) or there are insufficient 
resources available to allocate a queue pair (out of memory). The latter can 
happen if either 1) insufficient memory is available, or 2) no more physical 
memory can be registered with the device.

For more information on memory registration see the Open MPI FAQs at:
http://www.open-mpi.org/faq/?category=openfabrics#ib-locked-pages

Local host: node001
Local device:   qib0
Queue pair type:Reliable connected (RC)

---

i've also tried various nodes across the cluster (200+).  i think i ruled out 
errant switch (qlogic single 12800-120) problems, bad cables, and bad nodes.  
that's not to say they're may not be present, i've just not been able to find 
it ___
users mailing list
us...@open-mpi.org
Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
Link to this post: 
http://www.open-mpi.org/community/lists/users/2016/03/28717.php

Re: [OMPI users] locked memory and queue pairs

2016-03-16 Thread Cabral, Matias A

Hi Michael,

I may be missing some context, if you are using the qlogic cards you will 
always want to use the psm mtl (-mca pml cm -mca mtl psm) and not openib btl. 
As Tom suggest, confirm the limits are setup on every node: could it be the 
alltoall is reaching a node that "others" are not? Please share the command 
line and the error message.  

Thanks, 

_MAC

>> Begin forwarded message:
>> 
>> From: Michael Di Domenico 
>> Subject: Re: [OMPI users] locked memory and queue pairs
>> Date: March 16, 2016 at 11:32:01 AM EDT
>> To: Open MPI Users 
>> Reply-To: Open MPI Users 
>> 
>> On Thu, Mar 10, 2016 at 11:54 AM, Michael Di Domenico 
>>  wrote:
>>> when i try to run an openmpi job with >128 ranks (16 ranks per node) 
>>> using alltoall or alltoallv, i'm getting an error that the process 
>>> was unable to get a queue pair.
>>> 
>>> i've checked the max locked memory settings across my machines;
>>> 
>>> using ulimit -l in and outside of mpirun and they're all set to 
>>> unlimited pam modules to ensure pam_limits.so is loaded and working 
>>> the /etc/security/limits.conf is set for soft/hard mem to unlimited
>>> 
>>> i tried a couple of quick mpi config settings i could think of;
>>> 
>>> -mca mtl ^psm no affect
>>> -mca btl_openib_flags 1 no affect
>>> 
>>> the openmpi faq says to tweak some mtt values in /sys, but since i'm 
>>> not on mellanox that doesn't apply to me
>>> 
>>> the machines are rhel 6.7, kernel 2.6.32-573.12.1(with bundled 
>>> ofed), running on qlogic single-port infiniband cards, psm is 
>>> enabled
>>> 
>>> other collectives seem to run okay, it seems to only be alltoall 
>>> comms that fail and only at scale
>>> 
>>> i believe (but can't prove) that this worked at one point, but i 
>>> can't recall when i last tested it.  so it's reasonable to assume 
>>> that some change to the system is preventing this.
>>> 
>>> the question is, where should i start poking to find it?
>> 
>> bump?
>> ___
>> users mailing list
>> us...@open-mpi.org
>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
>> Link to this post: 
>> http://www.open-mpi.org/community/lists/users/2016/03/28713.php
>
>
>--
>Jeff Squyres
>jsquy...@cisco.com
>For corporate legal information go to: 
>http://www.cisco.com/web/about/doing_business/legal/cri/
>

Re: [OMPI users] Help Getting Started with Open MPI and PMIx and UCX

Re: [OMPI users] Help Getting Started with Open MPI and PMIx and UCX

Re: [OMPI users] Open MPI 4.0.0 - error with MPI_Send

Re: [OMPI users] Open MPI 4.0.0 - error with MPI_Send

Re: [OMPI users] Open MPI 4.0.0 - error with MPI_Send

Re: [OMPI users] Open MPI 4.0.0 - error with MPI_Send

Re: [OMPI users] A couple of general questions

Re: [OMPI users] A couple of general questions

Re: [OMPI users] OpenMPI with PSM on True Scale with OmniPath drivers

Re: [OMPI users] Received eager message(s) from an unknown process error on KNL

Re: [OMPI users] openmpi single node jobs using btl openib

Re: [OMPI users] Severe performance issue with PSM2 and single-node CP2K jobs

Re: [OMPI users] Open MPI Java Error

Re: [OMPI users] Error using hpcc benchmark

Re: [OMPI users] Error using hpcc benchmark

Re: [OMPI users] device failed to appear .. Connection timed out

Re: [OMPI users] Openmpi 2.0.1 build --with-psm2 failed on CentOS 7.2

Re: [OMPI users] Openmpi 2.0.1 build --with-psm2 failed on CentOS 7.2

Re: [OMPI users] Openmpi 2.0.1 build --with-psm2 failed on CentOS 7.2

Re: [OMPI users] MPI_Comm_spawn

Re: [OMPI users] runtime performance tuning for Intel OMA interconnect

Re: [OMPI users] runtime performance tuning for Intel OMA interconnect

Re: [OMPI users] runtime performance tuning for Intel OMA interconnect

Re: [OMPI users] PSM vs PSM2

Re: [OMPI users] locked memory and queue pairs

Re: [OMPI users] locked memory and queue pairs

Re: [OMPI users] locked memory and queue pairs

27 matches

Site Navigation

Mail list logo

Footer information