Re: [OMPI users] Error in file base/plm_base_launch_support.c: OPAL_HWLOC_TOPO

[r...@open-mpi.org]

More than likely the problem is the difference in hwloc versions - sounds like 
the topology to/from xml is different between the two versions, and the older 
one doesn’t understand the new one.

> On Jul 21, 2018, at 12:04 PM, Brian Smith  
> wrote:
> 
> Greetings,
> 
> I'm having trouble getting openmpi 2.1.2 to work when launching a
> process from debian 8 on a remote debian 9 host. To keep things simple
> in this example, I'm just launching date on the remote host.
> 
> deb8host$ mpirun -H deb9host date
> [deb8host:01552] [[32763,0],0] ORTE_ERROR_LOG: Error in file
> base/plm_base_launch_support.c at line 954
> 
> It works fine when executed from debian 9:
> deb9host$ mpirun -H deb8host date
> Sat Jul 21 13:40:43 CDT 2018
> 
> Also works when executed from debian 8 against debian 8:
> deb8host:~$ mpirun -H deb8host2 date
> Sat Jul 21 13:55:57 CDT 2018
> 
> The failure results from an error code returned by:
> opal_dss.unpack(buffer, , , OPAL_HWLOC_TOPO)
> 
> openmpi was built with the same configure flags on both hosts.
> 
>--prefix=$(PREFIX) \
>--with-verbs \
>--with-libfabric \
>--disable-silent-rules \
>--with-hwloc=/usr \
>--with-libltdl=/usr \
>--with-devel-headers \
>--with-slurm \
>--with-sge \
>--without-tm \
>--disable-heterogeneous \
>--with-contrib-vt-flags=--disable-iotrace \
>--sysconfdir=$(PREFIX)/etc \
>--libdir=$(PREFIX)/lib\
>--includedir=$(PREFIX)/include
> 
> 
> deb9host libhwloc and libhwloc-plugins is 1.11.5-1
> deb8host libhwloc and libhwloc-plugins is 1.10.0-3
> 
> I've been trying to debug this for the past few days and would
> appreciate any help on determining why this failure is occurring
> and/or resolving the problem.
> 
> -- 
> Brian T. Smith
> System Fabric Works
> Senior Technical Staff
> bsm...@systemfabricworks.com
> GPG Key: B3C2C7B73BA3CD7F
> ___
> users mailing list
> users@lists.open-mpi.org
> https://lists.open-mpi.org/mailman/listinfo/users

___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users

[MTT users] MTT docs now online

[r...@open-mpi.org]

We finally got the bugs out and the documentation for Python MTT implementation 
is now online at https://open-mpi.github.io/mtt 
.
It also picked up the Perl stuff, but we’ll just ignore that little detail :-)

Thanks to Akshaya Jagannadharao for all the hard work to make that happen!

Ralph

___
mtt-users mailing list
mtt-users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/mtt-users

Re: [OMPI users] Locking down TCP ports used

[r...@open-mpi.org]

I suspect the OOB is working just fine and you are seeing the TCP/btl opening 
the other ports. There are two TCP elements at work here: the OOB (which sends 
management messages between daemons) and the BTL (which handles the MPI 
traffic). In addition to what you provided, you also need to provide the 
following params:

btl_tcp_port_range_v4: The number of ports where the TCP BTL will try to bind.
  This parameter together with the port min, define a range of ports
  where Open MPI will open sockets

btl_tcp_port_min_v4: starting port to use

I can’t answer the question about #ports to open - will have to leave that to 
someone else
Ralph

> On Jul 7, 2018, at 6:34 AM, Adam Sylvester  wrote:
> 
> I'm using OpenMPI 2.1.0 on RHEL 7, communicating between ranks via TCP
> 
> I have a new cluster to install my application on with tightly-controlled 
> firewalls.  I can have them open up a range of TCP ports which MPI can 
> communicate over.  I thought I could force MPI to stick to a range of ports 
> via "--mca oob_tcp_static_ports startPort-endPort" but this doesn't seem to 
> be working; I still seem MPI opening up TCP ports outside of this range to 
> communicate.  I've also seen "--mca oob_tcp_dynamic_ports" on message boards; 
> I'm not sure what the difference is between these two but this flag doesn't 
> seem to do what I want either.
> 
> Is there a way to lock the TCP port range down?  As a general rule of thumb, 
> if I'm communicating between up to 50 instances on a 10 Gbps network moving 
> at several painful spots in the chain hundreds of GBs of data around, how 
> large should I make this port range (i.e. if Open MPI would normally open a 
> bunch of ports on each machine to improve the network transfer speed, I don't 
> want to slow it down by allowing it too narrow of a port range).  Just need a 
> rough order of magnitude - 10 ports, 100 ports, 1000 ports?
> 
> Thanks!
> -Adam
> ___
> users mailing list
> users@lists.open-mpi.org
> https://lists.open-mpi.org/mailman/listinfo/users

___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users

Re: [OMPI users] Verbose output for MPI

[r...@open-mpi.org]

Try adding PMIX_MCA_ptl_base_verbose=10 to your environment

> On Jul 4, 2018, at 8:51 AM, Maksym Planeta  
> wrote:
> 
> Thanks for quick response,
> 
> I tried this out and I do get more output: https://pastebin.com/JkXAYdM4. But 
> the line I need does not appear in the output.
> 
> On 04/07/18 17:38, Nathan Hjelm via users wrote:
>> --mca pmix_base_verbose 100
> -- 
> Regards,
> Maksym Planeta
> ___
> users mailing list
> users@lists.open-mpi.org
> https://lists.open-mpi.org/mailman/listinfo/users

___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users

Re: [OMPI users] Disable network interface selection

[r...@open-mpi.org]

> On Jun 22, 2018, at 8:25 PM, r...@open-mpi.org wrote:
> 
> 
> 
>> On Jun 22, 2018, at 7:31 PM, Gilles Gouaillardet 
>> mailto:gilles.gouaillar...@gmail.com>> wrote:
>> 
>> Carlos,
>> 
>> By any chance, could
>> 
>> mpirun—mca oob_tcp_if_exclude 192.168.100.0/24 <http://192.168.100.0/24> ...
>> 
>> work for you ?
>> 
>> Which Open MPI version are you running ?
>> 
>> 
>> IIRC, subnets are internally translated to interfaces, so that might be an 
>> issue if
>> the translation if made on the first host, and then the interface name is 
>> sent to the other hosts.
> 
> FWIW: we never send interface names to other hosts - just dot addresses

Should have clarified - when you specify an interface name for the MCA param, 
then it is the interface name that is transferred as that is the value of the 
MCA param. However, once we determine our address, we only transfer dot 
addresses between ourselves


> 
>> 
>> Cheers,
>> 
>> Gilles
>> 
>> On Saturday, June 23, 2018, carlos aguni > <mailto:aguni...@gmail.com>> wrote:
>> Hi all, 
>> 
>> I'm trying to run a code on 2 machines that has at least 2 network 
>> interfaces in it.
>> So I have them as described below:
>> 
>> compute01
>> compute02
>> ens3
>> 192.168.100.104/24 <http://192.168.100.104/24>   
>> 10.0.0.227/24 <http://10.0.0.227/24>
>> ens8
>> 10.0.0.228/24 <http://10.0.0.228/24> 
>> 172.21.1.128/24 <http://172.21.1.128/24>
>> ens9
>> 172.21.1.155/24 <http://172.21.1.155/24> 
>> ---
>> 
>> Issue is. When I execute `mpirun -n 2 -host compute01,compute02 hostname` on 
>> them what I get is the correct output after a very long delay..
>> 
>> What I've read so far is that OpenMPI performs a greedy algorithm on each 
>> interface that timeouts if it doesn't find the desired IP.
>> Then I saw here (https://www.open-mpi.org/faq/?category=tcp#tcp-selection 
>> <https://www.open-mpi.org/faq/?category=tcp#tcp-selection>) that I can run 
>> commands like:
>> `$ mpirun -n 2 --mca oob_tcp_if_include 10.0.0.0/24 <http://10.0.0.0/24> -n 
>> 2 -host compute01,compute02 hosname`
>> But this configuration doesn't reach the other host(s).
>> In the end I sometimes I get the same timeout.
>> 
>> So is there a way to let it to use the system's default route?
>> 
>> Regards,
>> Carlos.
>> ___
>> users mailing list
>> users@lists.open-mpi.org <mailto:users@lists.open-mpi.org>
>> https://lists.open-mpi.org/mailman/listinfo/users

___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users

Re: [OMPI users] Disable network interface selection

[r...@open-mpi.org]

> On Jun 22, 2018, at 7:31 PM, Gilles Gouaillardet 
>  wrote:
> 
> Carlos,
> 
> By any chance, could
> 
> mpirun—mca oob_tcp_if_exclude 192.168.100.0/24  ...
> 
> work for you ?
> 
> Which Open MPI version are you running ?
> 
> 
> IIRC, subnets are internally translated to interfaces, so that might be an 
> issue if
> the translation if made on the first host, and then the interface name is 
> sent to the other hosts.

FWIW: we never send interface names to other hosts - just dot addresses

> 
> Cheers,
> 
> Gilles
> 
> On Saturday, June 23, 2018, carlos aguni  > wrote:
> Hi all, 
> 
> I'm trying to run a code on 2 machines that has at least 2 network interfaces 
> in it.
> So I have them as described below:
> 
> compute01
> compute02
> ens3
> 192.168.100.104/24 
> 10.0.0.227/24 
> ens8
> 10.0.0.228/24   
> 172.21.1.128/24 
> ens9
> 172.21.1.155/24   
> ---
> 
> Issue is. When I execute `mpirun -n 2 -host compute01,compute02 hostname` on 
> them what I get is the correct output after a very long delay..
> 
> What I've read so far is that OpenMPI performs a greedy algorithm on each 
> interface that timeouts if it doesn't find the desired IP.
> Then I saw here (https://www.open-mpi.org/faq/?category=tcp#tcp-selection 
> ) that I can run 
> commands like:
> `$ mpirun -n 2 --mca oob_tcp_if_include 10.0.0.0/24  -n 2 
> -host compute01,compute02 hosname`
> But this configuration doesn't reach the other host(s).
> In the end I sometimes I get the same timeout.
> 
> So is there a way to let it to use the system's default route?
> 
> Regards,
> Carlos.
> ___
> users mailing list
> users@lists.open-mpi.org
> https://lists.open-mpi.org/mailman/listinfo/users

___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users

Re: [OMPI users] new core binding issues?

[r...@open-mpi.org]

Afraid I’m not familiar with that option, so I really don’t know :-(

> On Jun 22, 2018, at 10:13 AM, Noam Bernstein  
> wrote:
> 
>> On Jun 22, 2018, at 1:00 PM, r...@open-mpi.org <mailto:r...@open-mpi.org> 
>> wrote:
>> 
>> I suspect it is okay. Keep in mind that OMPI itself is starting multiple 
>> progress threads, so that is likely what you are seeing. The binding patter 
>> in the mpirun output looks correct as the default would be to map-by socket 
>> and you asked that we bind-to core.
> 
> I agree that the mpirun output makes sense - I’m more worried about the PSR 
> field in the output of ps.  Is it really consistent with what mpirun asked 
> for?
> 
>   
> Noam
> 
> 
> 
> 
> ||
> |U.S. NAVAL|
> |_RESEARCH_|
> LABORATORY
> 
> Noam Bernstein, Ph.D.
> Center for Materials Physics and Technology
> U.S. Naval Research Laboratory
> T +1 202 404 8628  F +1 202 404 7546
> https://www.nrl.navy.mil <https://www.nrl.navy.mil/>
> ___
> users mailing list
> users@lists.open-mpi.org
> https://lists.open-mpi.org/mailman/listinfo/users

___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users

Re: [OMPI users] new core binding issues?

[r...@open-mpi.org]

I suspect it is okay. Keep in mind that OMPI itself is starting multiple 
progress threads, so that is likely what you are seeing. The binding patter in 
the mpirun output looks correct as the default would be to map-by socket and 
you asked that we bind-to core.


> On Jun 22, 2018, at 9:33 AM, Noam Bernstein  
> wrote:
> 
> Hi - for the last couple of weeks, more or less since we did some kernel 
> updates, certain compute intensive MPI jobs have been behaving oddly as far 
> as their speed - bits that should be quite fast sometimes (but not 
> consistently) take a long time, and re-running sometimes fixes the issue, 
> sometimes not.  I’m starting to suspect core binding problems, which I worry 
> will be difficult to debug, so I hoped to get some feedback on whether my 
> observations are indeed suggesting that there’s something wrong with the core 
> binding.
> 
> I’m running withe CentOS 6 latest kernel (2.6.32-696.30.1.el6.x86_64), 
> OpenMPI 3.1.0, a dual cpu 8 core + HT intel Xeon node.  Code is compiled with 
> ifort, using “-mkl=sequential”, and just to be certain OMP_NUM_THREADS=1, so 
> there should be no OpenMP parallelism.
> 
> The main question is if I’m running 16 MPI tasks per node and look at the PSR 
> field from ps, should I get some simple sequence of numbers?
> 
> Here’s the beginning of the output report on the per-core binding I requested 
> from mpirun (—bind-to core)
> [compute-7-2:31036] MCW rank 0 bound to socket 0[core 0[hwt 0-1]]: 
> [BB/../../../../../../..][../../../../../../../..]
> [compute-7-2:31036] MCW rank 1 bound to socket 1[core 8[hwt 0-1]]: 
> [../../../../../../../..][BB/../../../../../../..]
> [compute-7-2:31036] MCW rank 2 bound to socket 0[core 1[hwt 0-1]]: 
> [../BB/../../../../../..][../../../../../../../..]
> [compute-7-2:31036] MCW rank 3 bound to socket 1[core 9[hwt 0-1]]: 
> [../../../../../../../..][../BB/../../../../../..]
> [compute-7-2:31036] MCW rank 4 bound to socket 0[core 2[hwt 0-1]]: 
> [../../BB/../../../../..][../../../../../../../..]
> [compute-7-2:31036] MCW rank 5 bound to socket 1[core 10[hwt 0-1]]: 
> [../../../../../../../..][../../BB/../../../../..]
> [compute-7-2:31036] MCW rank 6 bound to socket 0[core 3[hwt 0-1]]: 
> [../../../BB/../../../..][../../../../../../../..]
> 
> This is the PSR info from ps
>   PID PSR TTY  TIME CMD
> 31043   1 ?00:00:34 vasp.para.intel
> 31045   2 ?00:00:34 vasp.para.intel
> 31047   3 ?00:00:34 vasp.para.intel
> 31049   4 ?00:00:34 vasp.para.intel
> 31051   5 ?00:00:34 vasp.para.intel
> 31055   7 ?00:00:34 vasp.para.intel
> 31042   8 ?00:00:34 vasp.para.intel
> 31046  10 ?00:00:34 vasp.para.intel
> 31048  11 ?00:00:34 vasp.para.intel
> 31052  13 ?00:00:34 vasp.para.intel
> 31054  14 ?00:00:34 vasp.para.intel
> 31053  22 ?00:00:34 vasp.para.intel
> 31044  25 ?00:00:34 vasp.para.intel
> 31050  28 ?00:00:34 vasp.para.intel
> 31056  31 ?00:00:34 vasp.para.intel
> 
> Does this output look reasonable? For any sensible way I can think of to 
> enumerate the 32 virtual cores, those numbers don’t seem to correspond to one 
> mpi task per core. If this isn’t supposed to be giving meaningful output 
> given how openmpi does its binding, is there another tool that can tell me 
> what cores a running job is actually running on/bound to?
> 
> An additional bit of confusion is that "ps -mo pid,tid,fname,user,psr -p PID” 
> on one of those processes (which is supposed to be running without threaded 
> parallelism) reports 3 separate TID (which I think correspond to threads), 
> with 3 different PSR values, that seem stable during the run, but don’t have 
> any connection to one another (not P and P+1, or P and P+8, or P and P+16).
> 
> 
>   
> thanks,
>   
> Noam
> ___
> users mailing list
> users@lists.open-mpi.org
> https://lists.open-mpi.org/mailman/listinfo/users

___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users

Re: [OMPI users] Enforcing specific interface and subnet usage

[r...@open-mpi.org]

The OMPI cmd line converts "--mca ptl_tcp_remote_connections 1” to OMPI_MCA_ 
ptl_tcp_remote_connections, which is not recognized by PMIx. PMIx is looking 
for PMIX_MCA_ptl_tcp_remote_connections. The only way to set PMIx MCA params 
for the code embedded in OMPI is to put them in your environment


> On Jun 19, 2018, at 2:08 AM, Maksym Planeta  
> wrote:
> 
> But what about remote connections parameter? Why is it not set?
> 
> On 19/06/18 00:58, r...@open-mpi.org wrote:
>> I’m not entirely sure I understand what you are trying to do. The 
>> PMIX_SERVER_URI2 envar tells local clients how to connect to their local 
>> PMIx server (i.e., the OMPI daemon on that node). This is always done over 
>> the loopback device since it is a purely local connection that is never used 
>> for MPI messages.
>> I’m sure that the tcp/btl is using your indicated subnet as that would be 
>> used for internode messages.
> -- 
> Regards,
> Maksym Planeta
> 
> ___
> users mailing list
> users@lists.open-mpi.org
> https://lists.open-mpi.org/mailman/listinfo/users

___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users

Re: [OMPI users] Enforcing specific interface and subnet usage

[r...@open-mpi.org]

I’m not entirely sure I understand what you are trying to do. The 
PMIX_SERVER_URI2 envar tells local clients how to connect to their local PMIx 
server (i.e., the OMPI daemon on that node). This is always done over the 
loopback device since it is a purely local connection that is never used for 
MPI messages.

I’m sure that the tcp/btl is using your indicated subnet as that would be used 
for internode messages.


> On Jun 18, 2018, at 3:52 PM, Maksym Planeta  
> wrote:
> 
> Hello,
> 
> I want to force OpenMPI to use TCP and in particular use a particular subnet. 
> Unfortunately, I can't manage to do that.
> 
> Here is what I try:
> 
> $BIN/mpirun --mca pml ob1 --mca btl tcp,self --mca ptl_tcp_remote_connections 
> 1 --mca btl_tcp_if_include '10.233.0.0/19' -np 4  --oversubscribe -H 
> ib1n,ib2n bash -c 'echo $PMIX_SERVER_URI2'
> 
> The expected result would be a list of IP addresses in 10.233.0.0 subnet, but 
> instead I get this:
> 
> 2659516416.2;tcp4://127.0.0.1:46777
> 2659516416.2;tcp4://127.0.0.1:46777
> 2659516416.1;tcp4://127.0.0.1:45055
> 2659516416.1;tcp4://127.0.0.1:45055
> 
> Could you help me to debug this problem somehow?
> 
> The IP addresses are completely available in the desired subnet
> 
> $BIN/mpirun --mca pml ob1 --mca btl tcp,self  --mca 
> ptl_tcp_remote_connections 1 --mca btl_tcp_if_include '10.233.0.0/19' -np 4  
> --oversubscribe -H ib1n,ib2n ip addr show dev br0
> 
> Returns a set of bridges looking like:
> 
> 9: br0:  mtu 1500 qdisc noqueue state UP 
> group default qlen 1000
>link/ether 94:de:80:ba:37:e4 brd ff:ff:ff:ff:ff:ff
>inet 141.76.49.17/26 brd 141.76.49.63 scope global br0
>   valid_lft forever preferred_lft forever
>inet 10.233.0.82/19 scope global br0
>   valid_lft forever preferred_lft forever
>inet6 2002:8d4c:3001:48:40de:80ff:feba:37e4/64 scope global deprecated 
> mngtmpaddr dynamic 
>   valid_lft 59528sec preferred_lft 0sec
>inet6 fe80::96de:80ff:feba:37e4/64 scope link tentative dadfailed 
>   valid_lft forever preferred_lft forever
> 
> 
> What is more boggling is that if I attache with a debugger at 
> opal/mca/pmix/pmix3x/pmix/src/mca/ptl/tcp/ptl_tcp_components.c around line 
> 500 I see that mca_ptl_tcp_component.remote_connections is false. This means 
> that the way I set up component parameters is ignored.
> 
> -- 
> Regards,
> Maksym Planeta
> 
> ___
> users mailing list
> users@lists.open-mpi.org
> https://lists.open-mpi.org/mailman/listinfo/users

___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users

Re: [OMPI users] srun works, mpirun does not

[r...@open-mpi.org]

This is on an ARM processor? I suspect that is the root of the problems as we 
aren’t seeing anything like this elsewhere.


> On Jun 18, 2018, at 1:27 PM, Bennet Fauber  wrote:
> 
> If it's of any use, 3.0.0 seems to hang at
> 
> Making check in class
> make[2]: Entering directory `/tmp/build/openmpi-3.0.0/test/class'
> make  ompi_rb_tree opal_bitmap opal_hash_table opal_proc_table
> opal_tree opal_list opal_value_array opal_pointer_array opal_lifo
> opal_fifo
> make[3]: Entering directory `/tmp/build/openmpi-3.0.0/test/class'
> make[3]: `ompi_rb_tree' is up to date.
> make[3]: `opal_bitmap' is up to date.
> make[3]: `opal_hash_table' is up to date.
> make[3]: `opal_proc_table' is up to date.
> make[3]: `opal_tree' is up to date.
> make[3]: `opal_list' is up to date.
> make[3]: `opal_value_array' is up to date.
> make[3]: `opal_pointer_array' is up to date.
> make[3]: `opal_lifo' is up to date.
> make[3]: `opal_fifo' is up to date.
> make[3]: Leaving directory `/tmp/build/openmpi-3.0.0/test/class'
> make  check-TESTS
> make[3]: Entering directory `/tmp/build/openmpi-3.0.0/test/class'
> make[4]: Entering directory `/tmp/build/openmpi-3.0.0/test/class'
> 
> I have to interrupt it, but it's been many minutes, and usually these
> have not been behaving this way.
> 
> -- bennet
> 
> On Mon, Jun 18, 2018 at 4:21 PM Bennet Fauber  wrote:
>> 
>> No such luck.  If it matters, mpirun does seem to work with processes
>> on the local node that have no internal MPI code.  That is,
>> 
>> [bennet@cavium-hpc ~]$ mpirun -np 4 hello
>> Hello, ARM
>> Hello, ARM
>> Hello, ARM
>> Hello, ARM
>> 
>> but it fails with a similar error if run while a SLURM job is active; i.e.,
>> 
>> [bennet@cavium-hpc ~]$ mpirun hello
>> --
>> ORTE has lost communication with a remote daemon.
>> 
>>  HNP daemon   : [[26589,0],0] on node cavium-hpc
>>  Remote daemon: [[26589,0],1] on node cav01
>> 
>> This is usually due to either a failure of the TCP network
>> connection to the node, or possibly an internal failure of
>> the daemon itself. We cannot recover from this failure, and
>> therefore will terminate the job.
>> ----------
>> 
>> That makes sense, I guess.
>> 
>> I'll keep you posted as to what happens with 3.0.0 and downgrading SLURM.
>> 
>> 
>> Thanks,   -- bennet
>> 
>> 
>> On Mon, Jun 18, 2018 at 4:05 PM r...@open-mpi.org  wrote:
>>> 
>>> I doubt Slurm is the issue. For grins, lets try adding “--mca plm rsh” to 
>>> your mpirun cmd line and see if that works.
>>> 
>>> 
>>>> On Jun 18, 2018, at 12:57 PM, Bennet Fauber  wrote:
>>>> 
>>>> To eliminate possibilities, I removed all other versions of OpenMPI
>>>> from the system, and rebuilt using the same build script as was used
>>>> to generate the prior report.
>>>> 
>>>> [bennet@cavium-hpc bennet]$ ./ompi-3.1.0bd.sh
>>>> Checking compilers and things
>>>> OMPI is ompi
>>>> COMP_NAME is gcc_7_1_0
>>>> SRC_ROOT is /sw/arcts/centos7/src
>>>> PREFIX_ROOT is /sw/arcts/centos7
>>>> PREFIX is /sw/arcts/centos7/gcc_7_1_0/openmpi/3.1.0-bd
>>>> CONFIGURE_FLAGS are
>>>> COMPILERS are CC=gcc CXX=g++ FC=gfortran
>>>> 
>>>> Currently Loaded Modules:
>>>> 1) gcc/7.1.0
>>>> 
>>>> gcc (ARM-build-14) 7.1.0
>>>> Copyright (C) 2017 Free Software Foundation, Inc.
>>>> This is free software; see the source for copying conditions.  There is NO
>>>> warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
>>>> 
>>>> Using the following configure command
>>>> 
>>>> ./configure --prefix=/sw/arcts/centos7/gcc_7_1_0/openmpi/3.1.0-bd
>>>>  --mandir=/sw/arcts/centos7/gcc_7_1_0/openmpi/3.1.0-bd/share/man
>>>> --with-pmix=/opt/pmix/2.0.2 --with-libevent=external
>>>> --with-hwloc=external --with-slurm --disable-dlopen
>>>> --enable-debug  CC=gcc CXX=g++ FC=gfortran
>>>> 
>>>> The tar ball is
>>>> 
>>>> 2e783873f6b206aa71f745762fa15da5
>>>> /sw/arcts/centos7/src/ompi/openmpi-3.1.0.tar.gz
>>>> 
>>>> I still get
>>>> 
>>>> [bennet@cavium-hpc ~]$ salloc -N 1 --ntasks-per-node=24
>>>> salloc: Pending job allocation 165

Re: [OMPI users] srun works, mpirun does not

[r...@open-mpi.org]

I doubt Slurm is the issue. For grins, lets try adding “--mca plm rsh” to your 
mpirun cmd line and see if that works.


> On Jun 18, 2018, at 12:57 PM, Bennet Fauber  wrote:
> 
> To eliminate possibilities, I removed all other versions of OpenMPI
> from the system, and rebuilt using the same build script as was used
> to generate the prior report.
> 
> [bennet@cavium-hpc bennet]$ ./ompi-3.1.0bd.sh
> Checking compilers and things
> OMPI is ompi
> COMP_NAME is gcc_7_1_0
> SRC_ROOT is /sw/arcts/centos7/src
> PREFIX_ROOT is /sw/arcts/centos7
> PREFIX is /sw/arcts/centos7/gcc_7_1_0/openmpi/3.1.0-bd
> CONFIGURE_FLAGS are
> COMPILERS are CC=gcc CXX=g++ FC=gfortran
> 
> Currently Loaded Modules:
>  1) gcc/7.1.0
> 
> gcc (ARM-build-14) 7.1.0
> Copyright (C) 2017 Free Software Foundation, Inc.
> This is free software; see the source for copying conditions.  There is NO
> warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
> 
> Using the following configure command
> 
> ./configure --prefix=/sw/arcts/centos7/gcc_7_1_0/openmpi/3.1.0-bd
>   --mandir=/sw/arcts/centos7/gcc_7_1_0/openmpi/3.1.0-bd/share/man
> --with-pmix=/opt/pmix/2.0.2 --with-libevent=external
> --with-hwloc=external --with-slurm --disable-dlopen
> --enable-debug  CC=gcc CXX=g++ FC=gfortran
> 
> The tar ball is
> 
> 2e783873f6b206aa71f745762fa15da5
> /sw/arcts/centos7/src/ompi/openmpi-3.1.0.tar.gz
> 
> I still get
> 
> [bennet@cavium-hpc ~]$ salloc -N 1 --ntasks-per-node=24
> salloc: Pending job allocation 165
> salloc: job 165 queued and waiting for resources
> salloc: job 165 has been allocated resources
> salloc: Granted job allocation 165
> [bennet@cavium-hpc ~]$ srun ./test_mpi
> The sum = 0.866386
> Elapsed time is:  5.425549
> The sum = 0.866386
> Elapsed time is:  5.422826
> The sum = 0.866386
> Elapsed time is:  5.427676
> The sum = 0.866386
> Elapsed time is:  5.424928
> The sum = 0.866386
> Elapsed time is:  5.422060
> The sum = 0.866386
> Elapsed time is:  5.425431
> The sum = 0.866386
> Elapsed time is:  5.424350
> The sum = 0.866386
> Elapsed time is:  5.423037
> The sum = 0.866386
> Elapsed time is:  5.427727
> The sum = 0.866386
> Elapsed time is:  5.424922
> The sum = 0.866386
> Elapsed time is:  5.424279
> Total time is:  59.672992
> 
> [bennet@cavium-hpc ~]$ mpirun ./test_mpi
> --
> An ORTE daemon has unexpectedly failed after launch and before
> communicating back to mpirun. This could be caused by a number
> of factors, including an inability to create a connection back
> to mpirun due to a lack of common network interfaces and/or no
> route found between them. Please check network connectivity
> (including firewalls and network routing requirements).
> --
> 
> I reran with
> 
> [bennet@cavium-hpc ~]$ mpirun --mca plm_base_verbose 10 ./test_mpi
> 2>&1 | tee debug3.log
> 
> and the gzipped log is attached.
> 
> I thought to try it with a different test program, which spits the error
> [cavium-hpc.arc-ts.umich.edu:42853] [[58987,1],0] ORTE_ERROR_LOG: Not
> found in file base/ess_base_std_app.c at line 219
> [cavium-hpc.arc-ts.umich.edu:42854] [[58987,1],1] ORTE_ERROR_LOG: Not
> found in file base/ess_base_std_app.c at line 219
> --
> It looks like orte_init failed for some reason; your parallel process is
> likely to abort.  There are many reasons that a parallel process can
> fail during orte_init; some of which are due to configuration or
> environment problems.  This failure appears to be an internal failure;
> here's some additional information (which may only be relevant to an
> Open MPI developer):
> 
>  store DAEMON URI failed
>  --> Returned value Not found (-13) instead of ORTE_SUCCESS
> 
> 
> At one point, I am almost certain that OMPI mpirun did work, and I am
> at a loss to explain why it no longer does.
> 
> I have also tried the 3.1.1rc1 version.  I am now going to try 3.0.0,
> and we'll try downgrading SLURM to a prior version.
> 
> -- bennet
> 
> 
> -- bennetOn Mon, Jun 18, 2018 at 10:56 AM r...@open-mpi.org
>  wrote:
>> 
>> Hmmm...well, the error has changed from your initial report. Turning off the 
>> firewall was the solution to that problem.
>> 
>> This problem is different - it isn’t the orted that failed in the log you 
>> sent, but the application proc that couldn’t initialize. It looks like that 
>> app was compiled against some earlier version 

Re: [OMPI users] srun works, mpirun does not

[r...@open-mpi.org]

Hmmm...well, the error has changed from your initial report. Turning off the 
firewall was the solution to that problem.

This problem is different - it isn’t the orted that failed in the log you sent, 
but the application proc that couldn’t initialize. It looks like that app was 
compiled against some earlier version of OMPI? It is looking for something that 
no longer exists. I saw that you compiled it with a simple “gcc” instead of our 
wrapper compiler “mpicc” - any particular reason? My guess is that your compile 
picked up some older version of OMPI on the system.

Ralph


> On Jun 17, 2018, at 2:51 PM, Bennet Fauber  wrote:
> 
> I rebuilt with --enable-debug, then ran with
> 
> [bennet@cavium-hpc ~]$ salloc -N 1 --ntasks-per-node=24
> salloc: Pending job allocation 158
> salloc: job 158 queued and waiting for resources
> salloc: job 158 has been allocated resources
> salloc: Granted job allocation 158
> 
> [bennet@cavium-hpc ~]$ srun ./test_mpi
> The sum = 0.866386
> Elapsed time is:  5.426759
> The sum = 0.866386
> Elapsed time is:  5.424068
> The sum = 0.866386
> Elapsed time is:  5.426195
> The sum = 0.866386
> Elapsed time is:  5.426059
> The sum = 0.866386
> Elapsed time is:  5.423192
> The sum = 0.866386
> Elapsed time is:  5.426252
> The sum = 0.866386
> Elapsed time is:  5.425444
> The sum = 0.866386
> Elapsed time is:  5.423647
> The sum = 0.866386
> Elapsed time is:  5.426082
> The sum = 0.866386
> Elapsed time is:  5.425936
> The sum = 0.866386
> Elapsed time is:  5.423964
> Total time is:  59.677830
> 
> [bennet@cavium-hpc ~]$ mpirun --mca plm_base_verbose 10 ./test_mpi
> 2>&1 | tee debug2.log
> 
> The zipped debug log should be attached.
> 
> I did that after using systemctl to turn off the firewall on the login
> node from which the mpirun is executed, as well as on the host on
> which it runs.
> 
> [bennet@cavium-hpc ~]$ mpirun hostname
> --
> An ORTE daemon has unexpectedly failed after launch and before
> communicating back to mpirun. This could be caused by a number
> of factors, including an inability to create a connection back
> to mpirun due to a lack of common network interfaces and/or no
> route found between them. Please check network connectivity
> (including firewalls and network routing requirements).
> --
> 
> [bennet@cavium-hpc ~]$ squeue
> JOBID PARTITION NAME USER ST   TIME  NODES
> NODELIST(REASON)
>   158  standard bash   bennet  R  14:30  1 cav01
> [bennet@cavium-hpc ~]$ srun hostname
> cav01.arc-ts.umich.edu
> [ repeated 23 more times ]
> 
> As always, your help is much appreciated,
> 
> -- bennet
> 
> On Sun, Jun 17, 2018 at 1:06 PM r...@open-mpi.org  wrote:
>> 
>> Add --enable-debug to your OMPI configure cmd line, and then add --mca 
>> plm_base_verbose 10 to your mpirun cmd line. For some reason, the remote 
>> daemon isn’t starting - this will give you some info as to why.
>> 
>> 
>>> On Jun 17, 2018, at 9:07 AM, Bennet Fauber  wrote:
>>> 
>>> I have a compiled binary that will run with srun but not with mpirun.
>>> The attempts to run with mpirun all result in failures to initialize.
>>> I have tried this on one node, and on two nodes, with firewall turned
>>> on and with it off.
>>> 
>>> Am I missing some command line option for mpirun?
>>> 
>>> OMPI built from this configure command
>>> 
>>> $ ./configure --prefix=/sw/arcts/centos7/gcc_7_1_0/openmpi/3.1.0-b
>>> --mandir=/sw/arcts/centos7/gcc_7_1_0/openmpi/3.1.0-b/share/man
>>> --with-pmix=/opt/pmix/2.0.2 --with-libevent=external
>>> --with-hwloc=external --with-slurm --disable-dlopen CC=gcc CXX=g++
>>> FC=gfortran
>>> 
>>> All tests from `make check` passed, see below.
>>> 
>>> [bennet@cavium-hpc ~]$ mpicc --show
>>> gcc -I/sw/arcts/centos7/gcc_7_1_0/openmpi/3.1.0-b/include -pthread
>>> -L/opt/pmix/2.0.2/lib -Wl,-rpath -Wl,/opt/pmix/2.0.2/lib -Wl,-rpath
>>> -Wl,/sw/arcts/centos7/gcc_7_1_0/openmpi/3.1.0-b/lib
>>> -Wl,--enable-new-dtags
>>> -L/sw/arcts/centos7/gcc_7_1_0/openmpi/3.1.0-b/lib -lmpi
>>> 
>>> The test_mpi was compiled with
>>> 
>>> $ gcc -o test_mpi test_mpi.c -lm
>>> 
>>> This is the runtime library path
>>> 
>>> [bennet@cavium-hpc ~]$ echo $LD_LIBRARY_PATH
>>> /opt/slurm/lib64:/sw/arcts/centos7/gcc_7_1_0/openmpi/3.1.0-b/lib:/opt/arm/gcc-7.1.0_Generic-AArch64_RH

Re: [OMPI users] Fwd: srun works, mpirun does not

[r...@open-mpi.org]

Add --enable-debug to your OMPI configure cmd line, and then add --mca 
plm_base_verbose 10 to your mpirun cmd line. For some reason, the remote daemon 
isn’t starting - this will give you some info as to why.


> On Jun 17, 2018, at 9:07 AM, Bennet Fauber  wrote:
> 
> I have a compiled binary that will run with srun but not with mpirun.
> The attempts to run with mpirun all result in failures to initialize.
> I have tried this on one node, and on two nodes, with firewall turned
> on and with it off.
> 
> Am I missing some command line option for mpirun?
> 
> OMPI built from this configure command
> 
>  $ ./configure --prefix=/sw/arcts/centos7/gcc_7_1_0/openmpi/3.1.0-b
> --mandir=/sw/arcts/centos7/gcc_7_1_0/openmpi/3.1.0-b/share/man
> --with-pmix=/opt/pmix/2.0.2 --with-libevent=external
> --with-hwloc=external --with-slurm --disable-dlopen CC=gcc CXX=g++
> FC=gfortran
> 
> All tests from `make check` passed, see below.
> 
> [bennet@cavium-hpc ~]$ mpicc --show
> gcc -I/sw/arcts/centos7/gcc_7_1_0/openmpi/3.1.0-b/include -pthread
> -L/opt/pmix/2.0.2/lib -Wl,-rpath -Wl,/opt/pmix/2.0.2/lib -Wl,-rpath
> -Wl,/sw/arcts/centos7/gcc_7_1_0/openmpi/3.1.0-b/lib
> -Wl,--enable-new-dtags
> -L/sw/arcts/centos7/gcc_7_1_0/openmpi/3.1.0-b/lib -lmpi
> 
> The test_mpi was compiled with
> 
> $ gcc -o test_mpi test_mpi.c -lm
> 
> This is the runtime library path
> 
> [bennet@cavium-hpc ~]$ echo $LD_LIBRARY_PATH
> /opt/slurm/lib64:/sw/arcts/centos7/gcc_7_1_0/openmpi/3.1.0-b/lib:/opt/arm/gcc-7.1.0_Generic-AArch64_RHEL-7_aarch64-linux/lib64:/opt/arm/gcc-7.1.0_Generic-AArch64_RHEL-7_aarch64-linux/lib:/opt/slurm/lib64:/opt/pmix/2.0.2/lib:/sw/arcts/centos7/hpc-utils/lib
> 
> 
> These commands are given in exact sequence in which they were entered
> at a console.
> 
> [bennet@cavium-hpc ~]$ salloc -N 1 --ntasks-per-node=24
> salloc: Pending job allocation 156
> salloc: job 156 queued and waiting for resources
> salloc: job 156 has been allocated resources
> salloc: Granted job allocation 156
> 
> [bennet@cavium-hpc ~]$ mpirun ./test_mpi
> --
> An ORTE daemon has unexpectedly failed after launch and before
> communicating back to mpirun. This could be caused by a number
> of factors, including an inability to create a connection back
> to mpirun due to a lack of common network interfaces and/or no
> route found between them. Please check network connectivity
> (including firewalls and network routing requirements).
> --
> 
> [bennet@cavium-hpc ~]$ srun ./test_mpi
> The sum = 0.866386
> Elapsed time is:  5.425439
> The sum = 0.866386
> Elapsed time is:  5.427427
> The sum = 0.866386
> Elapsed time is:  5.422579
> The sum = 0.866386
> Elapsed time is:  5.424168
> The sum = 0.866386
> Elapsed time is:  5.423951
> The sum = 0.866386
> Elapsed time is:  5.422414
> The sum = 0.866386
> Elapsed time is:  5.427156
> The sum = 0.866386
> Elapsed time is:  5.424834
> The sum = 0.866386
> Elapsed time is:  5.425103
> The sum = 0.866386
> Elapsed time is:  5.422415
> The sum = 0.866386
> Elapsed time is:  5.422948
> Total time is:  59.668622
> 
> Thanks,-- bennet
> 
> 
> make check results
> --
> 
> make  check-TESTS
> make[3]: Entering directory `/tmp/build/openmpi-3.1.0/ompi/debuggers'
> make[4]: Entering directory `/tmp/build/openmpi-3.1.0/ompi/debuggers'
> PASS: predefined_gap_test
> PASS: predefined_pad_test
> SKIP: dlopen_test
> 
> Testsuite summary for Open MPI 3.1.0
> 
> # TOTAL: 3
> # PASS:  2
> # SKIP:  1
> # XFAIL: 0
> # FAIL:  0
> # XPASS: 0
> # ERROR: 0
> 
> [ elided ]
> PASS: atomic_cmpset_noinline
>- 5 threads: Passed
> PASS: atomic_cmpset_noinline
>- 8 threads: Passed
> 
> Testsuite summary for Open MPI 3.1.0
> 
> # TOTAL: 8
> # PASS:  8
> # SKIP:  0
> # XFAIL: 0
> # FAIL:  0
> # XPASS: 0
> # ERROR: 0
> 
> [ elided ]
> make[4]: Entering directory `/tmp/build/openmpi-3.1.0/test/class'
> PASS: ompi_rb_tree
> PASS: opal_bitmap
> PASS: opal_hash_table
> PASS: opal_proc_table
> PASS: opal_tree
> PASS: opal_list
> PASS: opal_value_array
> PASS: opal_pointer_array
> PASS: opal_lifo
> PASS: opal_fifo
> 
> Testsuite summary for Open MPI 3.1.0
> 
> # TOTAL: 10
> # PASS:  10
> # SKIP:  0
> # XFAIL: 0
> # FAIL:  0
> # XPASS: 0
> # ERROR: 0
> 

Re: [OMPI users] Fwd: OpenMPI 3.1.0 on aarch64

[r...@open-mpi.org]

> On Jun 8, 2018, at 8:10 AM, Bennet Fauber  wrote:
> 
> Further testing shows that it was the failure to find the hwloc-devel files 
> that seems to be the cause of the failure.  I compiled and ran without the 
> additional configure flags, and it still seems to work.
> 
> I think it issued a two-line warning about this.  Is that something that 
> should result in an error if --with-hwloc=external is specified but not 
> found?  Just a thought.

Yes - that is a bug in our configury. It should have immediately error’d out.

> 
> My immediate problem is solved. Thanks very much Ralph and Artem for your 
> help!
> 
> -- bennet
> 
> 
> On Thu, Jun 7, 2018 at 11:06 AM r...@open-mpi.org <mailto:r...@open-mpi.org> 
> mailto:r...@open-mpi.org>> wrote:
> Odd - Artem, do you have any suggestions?
> 
> > On Jun 7, 2018, at 7:41 AM, Bennet Fauber  > <mailto:ben...@umich.edu>> wrote:
> > 
> > Thanks, Ralph,
> > 
> > I just tried it with
> > 
> >srun --mpi=pmix_v2 ./test_mpi
> > 
> > and got these messages
> > 
> > 
> > srun: Step created for job 89
> > [cav02.arc-ts.umich.edu:92286 <http://cav02.arc-ts.umich.edu:92286>] PMIX 
> > ERROR: OUT-OF-RESOURCE in file
> > client/pmix_client.c at line 234
> > [cav02.arc-ts.umich.edu:92286 <http://cav02.arc-ts.umich.edu:92286>] OPAL 
> > ERROR: Error in file
> > pmix2x_client.c at line 109
> > [cav02.arc-ts.umich.edu:92287 <http://cav02.arc-ts.umich.edu:92287>] PMIX 
> > ERROR: OUT-OF-RESOURCE in file
> > client/pmix_client.c at line 234
> > [cav02.arc-ts.umich.edu:92287 <http://cav02.arc-ts.umich.edu:92287>] OPAL 
> > ERROR: Error in file
> > pmix2x_client.c at line 109
> > --
> > The application appears to have been direct launched using "srun",
> > but OMPI was not built with SLURM's PMI support and therefore cannot
> > execute. There are several options for building PMI support under
> > SLURM, depending upon the SLURM version you are using:
> > 
> >  version 16.05 or later: you can use SLURM's PMIx support. This
> >  requires that you configure and build SLURM --with-pmix.
> > 
> >  Versions earlier than 16.05: you must use either SLURM's PMI-1 or
> >  PMI-2 support. SLURM builds PMI-1 by default, or you can manually
> >  install PMI-2. You must then build Open MPI using --with-pmi pointing
> >  to the SLURM PMI library location.
> > 
> > Please configure as appropriate and try again.
> > --
> > 
> > 
> > Just to be complete, I checked the library path,
> > 
> > 
> > $ ldconfig -p | egrep 'slurm|pmix'
> >libpmi2.so.1 (libc6,AArch64) => /opt/pmix/2.0.2/lib/libpmi2.so.1
> >libpmi2.so (libc6,AArch64) => /opt/pmix/2.0.2/lib/libpmi2.so
> >libpmix.so.2 (libc6,AArch64) => /opt/pmix/2.0.2/lib/libpmix.so.2
> >libpmix.so (libc6,AArch64) => /opt/pmix/2.0.2/lib/libpmix.so
> >libpmi.so.1 (libc6,AArch64) => /opt/pmix/2.0.2/lib/libpmi.so.1
> >libpmi.so (libc6,AArch64) => /opt/pmix/2.0.2/lib/libpmi.so
> > 
> > 
> > and libpmi* does appear there.
> > 
> > 
> > I also tried explicitly listing the slurm directory from the slurm
> > library installation in LD_LIBRARY_PATH, just in case it wasn't
> > traversing correctly.  that is, both
> > 
> > $ echo $LD_LIBRARY_PATH
> > /sw/arcts/centos7/gcc_7_1_0/openmpi/3.1.0/lib:/opt/arm/gcc-7.1.0_Generic-AArch64_RHEL-7_aarch64-linux/lib64:/opt/arm/gcc-7.1.0_Generic-AArch64_RHEL-7_aarch64-linux/lib:/opt/slurm/lib64:/sw/arcts/centos7/hpc-utils/lib
> > 
> > and
> > 
> > $ echo $LD_LIBRARY_PATH
> > /opt/slurm/lib64/slurm:/opt/pmix/2.0.2/lib:/sw/arcts/centos7/gcc_7_1_0/openmpi/3.1.0/lib:/opt/arm/gcc-7.1.0_Generic-AArch64_RHEL-7_aarch64-linux/lib64:/opt/arm/gcc-7.1.0_Generic-AArch64_RHEL-7_aarch64-linux/lib:/opt/slurm/lib64:/sw/arcts/centos7/hpc-utils/lib
> > 
> > 
> > I don't have a saved build log, but I can rebuild this and save the
> > build logs, in case any information in those logs would help.
> > 
> > I will also mention that we have, in the past, used the
> > --disable-dlopen and --enable-shared flags, which we did not use here.
> > Just in case that makes any difference.
> > 
> > -- bennet
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > On Thu, Jun 7, 2018 at 10:01 AM, r...@open-mpi.org 
> > <mailto:r...@open-mpi.org> m

Re: [OMPI users] Fwd: OpenMPI 3.1.0 on aarch64

[r...@open-mpi.org]

Odd - Artem, do you have any suggestions?

> On Jun 7, 2018, at 7:41 AM, Bennet Fauber  wrote:
> 
> Thanks, Ralph,
> 
> I just tried it with
> 
>srun --mpi=pmix_v2 ./test_mpi
> 
> and got these messages
> 
> 
> srun: Step created for job 89
> [cav02.arc-ts.umich.edu:92286] PMIX ERROR: OUT-OF-RESOURCE in file
> client/pmix_client.c at line 234
> [cav02.arc-ts.umich.edu:92286] OPAL ERROR: Error in file
> pmix2x_client.c at line 109
> [cav02.arc-ts.umich.edu:92287] PMIX ERROR: OUT-OF-RESOURCE in file
> client/pmix_client.c at line 234
> [cav02.arc-ts.umich.edu:92287] OPAL ERROR: Error in file
> pmix2x_client.c at line 109
> --
> The application appears to have been direct launched using "srun",
> but OMPI was not built with SLURM's PMI support and therefore cannot
> execute. There are several options for building PMI support under
> SLURM, depending upon the SLURM version you are using:
> 
>  version 16.05 or later: you can use SLURM's PMIx support. This
>  requires that you configure and build SLURM --with-pmix.
> 
>  Versions earlier than 16.05: you must use either SLURM's PMI-1 or
>  PMI-2 support. SLURM builds PMI-1 by default, or you can manually
>  install PMI-2. You must then build Open MPI using --with-pmi pointing
>  to the SLURM PMI library location.
> 
> Please configure as appropriate and try again.
> --
> 
> 
> Just to be complete, I checked the library path,
> 
> 
> $ ldconfig -p | egrep 'slurm|pmix'
>libpmi2.so.1 (libc6,AArch64) => /opt/pmix/2.0.2/lib/libpmi2.so.1
>libpmi2.so (libc6,AArch64) => /opt/pmix/2.0.2/lib/libpmi2.so
>libpmix.so.2 (libc6,AArch64) => /opt/pmix/2.0.2/lib/libpmix.so.2
>libpmix.so (libc6,AArch64) => /opt/pmix/2.0.2/lib/libpmix.so
>libpmi.so.1 (libc6,AArch64) => /opt/pmix/2.0.2/lib/libpmi.so.1
>libpmi.so (libc6,AArch64) => /opt/pmix/2.0.2/lib/libpmi.so
> 
> 
> and libpmi* does appear there.
> 
> 
> I also tried explicitly listing the slurm directory from the slurm
> library installation in LD_LIBRARY_PATH, just in case it wasn't
> traversing correctly.  that is, both
> 
> $ echo $LD_LIBRARY_PATH
> /sw/arcts/centos7/gcc_7_1_0/openmpi/3.1.0/lib:/opt/arm/gcc-7.1.0_Generic-AArch64_RHEL-7_aarch64-linux/lib64:/opt/arm/gcc-7.1.0_Generic-AArch64_RHEL-7_aarch64-linux/lib:/opt/slurm/lib64:/sw/arcts/centos7/hpc-utils/lib
> 
> and
> 
> $ echo $LD_LIBRARY_PATH
> /opt/slurm/lib64/slurm:/opt/pmix/2.0.2/lib:/sw/arcts/centos7/gcc_7_1_0/openmpi/3.1.0/lib:/opt/arm/gcc-7.1.0_Generic-AArch64_RHEL-7_aarch64-linux/lib64:/opt/arm/gcc-7.1.0_Generic-AArch64_RHEL-7_aarch64-linux/lib:/opt/slurm/lib64:/sw/arcts/centos7/hpc-utils/lib
> 
> 
> I don't have a saved build log, but I can rebuild this and save the
> build logs, in case any information in those logs would help.
> 
> I will also mention that we have, in the past, used the
> --disable-dlopen and --enable-shared flags, which we did not use here.
> Just in case that makes any difference.
> 
> -- bennet
> 
> 
> 
> 
> 
> 
> 
> On Thu, Jun 7, 2018 at 10:01 AM, r...@open-mpi.org  wrote:
>> I think you need to set your MPIDefault to pmix_v2 since you are using a 
>> PMIx v2 library
>> 
>> 
>>> On Jun 7, 2018, at 6:25 AM, Bennet Fauber  wrote:
>>> 
>>> Hi, Ralph,
>>> 
>>> Thanks for the reply, and sorry for the missing information.  I hope
>>> this fills in the picture better.
>>> 
>>> $ srun --version
>>> slurm 17.11.7
>>> 
>>> $ srun --mpi=list
>>> srun: MPI types are...
>>> srun: pmix_v2
>>> srun: openmpi
>>> srun: none
>>> srun: pmi2
>>> srun: pmix
>>> 
>>> We have pmix configured as the default in /opt/slurm/etc/slurm.conf
>>> 
>>>   MpiDefault=pmix
>>> 
>>> and on the x86_64 system configured the same way, a bare 'srun
>>> ./test_mpi' is sufficient and runs.
>>> 
>>> I have tried all of the following srun variations with no joy
>>> 
>>> 
>>> srun ./test_mpi
>>> srun --mpi=pmix ./test_mpi
>>> srun --mpi=pmi2 ./test_mpi
>>> srun --mpi=openmpi ./test_mpi
>>> 
>>> 
>>> I believe we are using the spec files that come with both pmix and
>>> with slurm, and the following to build the .rpm files used at
>>> installation
>>> 
>>> 
>>> $ rpmbuild --define '_prefix /opt/pmix/2.0.2' \
>>>   -ba pmix-2.0.2.spec
>>> 
>>&

Re: [OMPI users] Fwd: OpenMPI 3.1.0 on aarch64

[r...@open-mpi.org]

I think you need to set your MPIDefault to pmix_v2 since you are using a PMIx 
v2 library


> On Jun 7, 2018, at 6:25 AM, Bennet Fauber  wrote:
> 
> Hi, Ralph,
> 
> Thanks for the reply, and sorry for the missing information.  I hope
> this fills in the picture better.
> 
> $ srun --version
> slurm 17.11.7
> 
> $ srun --mpi=list
> srun: MPI types are...
> srun: pmix_v2
> srun: openmpi
> srun: none
> srun: pmi2
> srun: pmix
> 
> We have pmix configured as the default in /opt/slurm/etc/slurm.conf
> 
>MpiDefault=pmix
> 
> and on the x86_64 system configured the same way, a bare 'srun
> ./test_mpi' is sufficient and runs.
> 
> I have tried all of the following srun variations with no joy
> 
> 
> srun ./test_mpi
> srun --mpi=pmix ./test_mpi
> srun --mpi=pmi2 ./test_mpi
> srun --mpi=openmpi ./test_mpi
> 
> 
> I believe we are using the spec files that come with both pmix and
> with slurm, and the following to build the .rpm files used at
> installation
> 
> 
> $ rpmbuild --define '_prefix /opt/pmix/2.0.2' \
>-ba pmix-2.0.2.spec
> 
> $ rpmbuild --define '_prefix /opt/slurm' \
>--define '_with-pmix --with-pmix=/opt/pmix/2.0.2' \
>-ta slurm-17.11.7.tar.bz2
> 
> 
> I did use the '--with-pmix=/opt/pmix/2.0.2' option when building OpenMPI.
> 
> 
> In case it helps, we have these libraries on the aarch64 in
> /opt/slurm/lib64/slurm/mpi*
> 
> -rwxr-xr-x 1 root root 257288 May 30 15:27 /opt/slurm/lib64/slurm/mpi_none.so
> -rwxr-xr-x 1 root root 257240 May 30 15:27 
> /opt/slurm/lib64/slurm/mpi_openmpi.so
> -rwxr-xr-x 1 root root 668808 May 30 15:27 /opt/slurm/lib64/slurm/mpi_pmi2.so
> lrwxrwxrwx 1 root root 16 Jun  1 08:38
> /opt/slurm/lib64/slurm/mpi_pmix.so -> ./mpi_pmix_v2.so
> -rwxr-xr-x 1 root root 841312 May 30 15:27 
> /opt/slurm/lib64/slurm/mpi_pmix_v2.so
> 
> and on the x86_64, where it runs, we have a comparable list,
> 
> -rwxr-xr-x 1 root root 193192 May 30 15:20 /opt/slurm/lib64/slurm/mpi_none.so
> -rwxr-xr-x 1 root root 193192 May 30 15:20 
> /opt/slurm/lib64/slurm/mpi_openmpi.so
> -rwxr-xr-x 1 root root 622848 May 30 15:20 /opt/slurm/lib64/slurm/mpi_pmi2.so
> lrwxrwxrwx 1 root root 16 Jun  1 08:32
> /opt/slurm/lib64/slurm/mpi_pmix.so -> ./mpi_pmix_v2.so
> -rwxr-xr-x 1 root root 828232 May 30 15:20 
> /opt/slurm/lib64/slurm/mpi_pmix_v2.so
> 
> 
> Let me know if anything else would be helpful.
> 
> Thanks,-- bennet
> 
> On Thu, Jun 7, 2018 at 8:56 AM, r...@open-mpi.org  wrote:
>> You didn’t show your srun direct launch cmd line or what version of Slurm is 
>> being used (and how it was configured), so I can only provide some advice. 
>> If you want to use PMIx, then you have to do two things:
>> 
>> 1. Slurm must be configured to use PMIx - depending on the version, that 
>> might be there by default in the rpm
>> 
>> 2. you have to tell srun to use the pmix plugin (IIRC you add --mpi pmix to 
>> the cmd line - you should check that)
>> 
>> If your intent was to use Slurm’s PMI-1 or PMI-2, then you need to configure 
>> OMPI --with-pmi=
>> 
>> Ralph
>> 
>> 
>>> On Jun 7, 2018, at 5:21 AM, Bennet Fauber  wrote:
>>> 
>>> We are trying out MPI on an aarch64 cluster.
>>> 
>>> Our system administrators installed SLURM and PMIx 2.0.2 from .rpm.
>>> 
>>> I compiled OpenMPI using the ARM distributed gcc/7.1.0 using the
>>> configure flags shown in this snippet from the top of config.log
>>> 
>>> It was created by Open MPI configure 3.1.0, which was
>>> generated by GNU Autoconf 2.69.  Invocation command line was
>>> 
>>> $ ./configure --prefix=/sw/arcts/centos7/gcc_7_1_0/openmpi/3.1.0
>>> --mandir=/sw/arcts/centos7/gcc_7_1_0/openmpi/3.1.0/share/man
>>> --with-pmix=/opt/pmix/2.0.2 --with-libevent=external
>>> --with-hwloc=external --with-slurm CC=gcc CXX=g++ FC=gfortran
>>> 
>>> ## - ##
>>> ## Platform. ##
>>> ## - ##
>>> 
>>> hostname = cavium-hpc.arc-ts.umich.edu
>>> uname -m = aarch64
>>> uname -r = 4.11.0-45.4.1.el7a.aarch64
>>> uname -s = Linux
>>> uname -v = #1 SMP Fri Feb 2 17:11:57 UTC 2018
>>> 
>>> /usr/bin/uname -p = aarch64
>>> 
>>> 
>>> It checks for pmi and reports it found,
>>> 
>>> 
>>> configure:12680: checking if user requested external PMIx
>>> support(/opt/pmix/2.0.2)
>>> configure:12690: result: yes
>>> configure:12701: checking --with-external-pmix value
>>> configure:12725: r

Re: [OMPI users] Fwd: OpenMPI 3.1.0 on aarch64

[r...@open-mpi.org]

You didn’t show your srun direct launch cmd line or what version of Slurm is 
being used (and how it was configured), so I can only provide some advice. If 
you want to use PMIx, then you have to do two things:

1. Slurm must be configured to use PMIx - depending on the version, that might 
be there by default in the rpm

2. you have to tell srun to use the pmix plugin (IIRC you add --mpi pmix to the 
cmd line - you should check that)

If your intent was to use Slurm’s PMI-1 or PMI-2, then you need to configure 
OMPI --with-pmi=

Ralph


> On Jun 7, 2018, at 5:21 AM, Bennet Fauber  wrote:
> 
> We are trying out MPI on an aarch64 cluster.
> 
> Our system administrators installed SLURM and PMIx 2.0.2 from .rpm.
> 
> I compiled OpenMPI using the ARM distributed gcc/7.1.0 using the
> configure flags shown in this snippet from the top of config.log
> 
> It was created by Open MPI configure 3.1.0, which was
> generated by GNU Autoconf 2.69.  Invocation command line was
> 
>  $ ./configure --prefix=/sw/arcts/centos7/gcc_7_1_0/openmpi/3.1.0
> --mandir=/sw/arcts/centos7/gcc_7_1_0/openmpi/3.1.0/share/man
> --with-pmix=/opt/pmix/2.0.2 --with-libevent=external
> --with-hwloc=external --with-slurm CC=gcc CXX=g++ FC=gfortran
> 
> ## - ##
> ## Platform. ##
> ## - ##
> 
> hostname = cavium-hpc.arc-ts.umich.edu
> uname -m = aarch64
> uname -r = 4.11.0-45.4.1.el7a.aarch64
> uname -s = Linux
> uname -v = #1 SMP Fri Feb 2 17:11:57 UTC 2018
> 
> /usr/bin/uname -p = aarch64
> 
> 
> It checks for pmi and reports it found,
> 
> 
> configure:12680: checking if user requested external PMIx
> support(/opt/pmix/2.0.2)
> configure:12690: result: yes
> configure:12701: checking --with-external-pmix value
> configure:12725: result: sanity check ok (/opt/pmix/2.0.2/include)
> configure:12768: checking libpmix.* in /opt/pmix/2.0.2/lib64
> configure:12774: checking libpmix.* in /opt/pmix/2.0.2/lib
> configure:12794: checking PMIx version
> configure:12804: result: version file found
> 
> 
> It fails on the test for PMIx 3, which is expected, but then reports
> 
> 
> configure:12843: checking version 2x
> configure:12861: gcc -E -I/opt/pmix/2.0.2/include  conftest.c
> configure:12861: $? = 0
> configure:12862: result: found
> 
> 
> I have a small, test MPI program that I run, and it runs when run with
> mpirun using mpirun.  The processes running on the first node of a two
> node job are
> 
> 
> [bennet@cav02 ~]$ ps -ef | grep bennet | egrep 'test_mpi|srun'
> 
> bennet   20340 20282  0 08:04 ?00:00:00 mpirun ./test_mpi
> 
> bennet   20346 20340  0 08:04 ?00:00:00 srun
> --ntasks-per-node=1 --kill-on-bad-exit --cpu_bind=none --nodes=1
> --nodelist=cav03 --ntasks=1 orted -mca ess "slurm" -mca ess_base_jobid
> "3609657344" -mca ess_base_vpid "1" -mca ess_base_num_procs "2" -mca
> orte_node_regex "cav[2:2-3]@0(2)" -mca orte_hnp_uri
> "3609657344.0;tcp://10.242.15.36:58681"
> 
> bennet   20347 20346  0 08:04 ?00:00:00 srun
> --ntasks-per-node=1 --kill-on-bad-exit --cpu_bind=none --nodes=1
> --nodelist=cav03 --ntasks=1 orted -mca ess "slurm" -mca ess_base_jobid
> "3609657344" -mca ess_base_vpid "1" -mca ess_base_num_procs "2" -mca
> orte_node_regex "cav[2:2-3]@0(2)" -mca orte_hnp_uri
> "3609657344.0;tcp://10.242.15.36:58681"
> 
> bennet   20352 20340 98 08:04 ?00:01:50 ./test_mpi
> 
> bennet   20353 20340 98 08:04 ?00:01:50 ./test_mpi
> 
> 
> However, when I run it using srun directly, I get the following output:
> 
> 
> srun: Step created for job 87
> [cav02.arc-ts.umich.edu:19828] OPAL ERROR: Not initialized in file
> pmix2x_client.c at line 109
> --
> The application appears to have been direct launched using "srun",
> but OMPI was not built with SLURM's PMI support and therefore cannot
> execute. There are several options for building PMI support under
> SLURM, depending upon the SLURM version you are using:
> 
>  version 16.05 or later: you can use SLURM's PMIx support. This
>  requires that you configure and build SLURM --with-pmix.
> 
>  Versions earlier than 16.05: you must use either SLURM's PMI-1 or
>  PMI-2 support. SLURM builds PMI-1 by default, or you can manually
>  install PMI-2. You must then build Open MPI using --with-pmi pointing
>  to the SLURM PMI library location.
> 
> Please configure as appropriate and try again.
> --
> *** An error occurred in MPI_Init
> *** on a NULL communicator
> *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
> ***and potentially your MPI job)
> [cav02.arc-ts.umich.edu:19828] Local abort before MPI_INIT completed
> completed successfully, but am not able to aggregate error messages,
> and not able to guarantee that all other processes were killed!
> 
> 
> Using the same scheme to set this up on x86_64 worked, and I am taking
> installation parameters, test files, and job 

Re: [OMPI users] --oversubscribe option

[r...@open-mpi.org]

I’m not entirely sure what you are asking here. If you use oversubscribe, we do 
not bind your processes and you suffer some performance penalty for it. If you 
want to run one process/thread and retain binding, then do not use 
--oversubscribe and instead use --use-hwthread-cpus


> On Jun 6, 2018, at 3:06 AM, Mahmood Naderan  wrote:
> 
> Hi,
> On a Ryzen 1800x which has 8 cores and 16 threads, when I run "mpirun -np 16 
> lammps..." I get an error that there is not enough slot. It seems that 
> --oversubscribe option will fix that.
> 
> Odd thing is that when I run "mpirun -np 8 lammps" it takes about 46 minutes 
> to complete the job while with "mpirun --oversubscribe -np 16 lammps" it 
> takes about 39 minutes.
> 
> I want to be sure that I "-np 16" uses are logical cores. Is that confirmed 
> with --oversubscribe?
> 
> Regards,
> Mahmood
> 
> 
> ___
> users mailing list
> users@lists.open-mpi.org
> https://lists.open-mpi.org/mailman/listinfo/users

___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users

Re: [OMPI users] A hang in Rmpi at PMIx_Disconnect

[r...@open-mpi.org]

Yes, that does sound like a bug - the #connects must equal the #disconnects.


> On Jun 4, 2018, at 1:17 PM, marcin.krotkiewski  
> wrote:
> 
> huh. This code also runs, but it also only displays 4 connect / disconnect 
> messages. I should add that the test R script shows 4 connect, but 8 
> disconnect messages. Looks like a bug to me, but where? I guess we will try 
> to contact R forums and ask there.
> Bennet: I tried to use doMPI + startMPIcluster / closeCluster. In this case I 
> get a warning about fork being used:
> 
> --
> A process has executed an operation involving a call to the
> "fork()" system call to create a child process.  Open MPI is currently
> operating in a condition that could result in memory corruption or
> other system errors; your job may hang, crash, or produce silent
> data corruption.  The use of fork() (or system() or other calls that
> create child processes) is strongly discouraged.
> 
> The process that invoked fork was:
> 
>   Local host:  [[36000,2],1] (PID 23617)
> 
> If you are *absolutely sure* that your application will successfully
> and correctly survive a call to fork(), you may disable this warning
> by setting the mpi_warn_on_fork MCA parameter to 0.
> --
> And the process hangs as well - no change.
> Marcin
> 
> 
> On 06/04/2018 05:27 PM, r...@open-mpi.org <mailto:r...@open-mpi.org> wrote:
>> It might call disconnect more than once if it creates multiple 
>> communicators. Here’s another test case for that behavior:
>> 
>> 
>> 
>> 
>> 
>>> On Jun 4, 2018, at 7:08 AM, Bennet Fauber  
>>> <mailto:ben...@umich.edu> wrote:
>>> 
>>> Just out of curiosity, but would using Rmpi and/or doMPI help in any way?
>>> 
>>> -- bennet
>>> 
>>> 
>>> On Mon, Jun 4, 2018 at 10:00 AM, marcin.krotkiewski
>>>  <mailto:marcin.krotkiew...@gmail.com> wrote:
>>>> Thanks, Ralph!
>>>> 
>>>> Your code finishes normally, I guess then the reason might be lying in R.
>>>> Running the R code with -mca pmix_base_verbose 1 i see that each rank calls
>>>> ext2x:client disconnect twice (each PID prints the line twice)
>>>> 
>>>> [...]
>>>>3 slaves are spawned successfully. 0 failed.
>>>> [localhost.localdomain:11659] ext2x:client disconnect
>>>> [localhost.localdomain:11661] ext2x:client disconnect
>>>> [localhost.localdomain:11658] ext2x:client disconnect
>>>> [localhost.localdomain:11646] ext2x:client disconnect
>>>> [localhost.localdomain:11658] ext2x:client disconnect
>>>> [localhost.localdomain:11659] ext2x:client disconnect
>>>> [localhost.localdomain:11661] ext2x:client disconnect
>>>> [localhost.localdomain:11646] ext2x:client disconnect
>>>> 
>>>> In your example it's only called once per process.
>>>> 
>>>> Do you have any suspicion where the second call comes from? Might this be
>>>> the reason for the hang?
>>>> 
>>>> Thanks!
>>>> 
>>>> Marcin
>>>> 
>>>> 
>>>> On 06/04/2018 03:16 PM, r...@open-mpi.org <mailto:r...@open-mpi.org> wrote:
>>>> 
>>>> Try running the attached example dynamic code - if that works, then it
>>>> likely is something to do with how R operates.
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> On Jun 4, 2018, at 3:43 AM, marcin.krotkiewski
>>>>  <mailto:marcin.krotkiew...@gmail.com> wrote:
>>>> 
>>>> Hi,
>>>> 
>>>> I have some problems running R + Rmpi with OpenMPI 3.1.0 + PMIx 2.1.1. A
>>>> simple R script, which starts a few tasks, hangs at the end on diconnect.
>>>> Here is the script:
>>>> 
>>>> library(parallel)
>>>> numWorkers <- as.numeric(Sys.getenv("SLURM_NTASKS")) - 1
>>>> myCluster <- makeCluster(numWorkers, type = "MPI")
>>>> stopCluster(myCluster)
>>>> 
>>>> And here is how I run it:
>>>> 
>>>> SLURM_NTASKS=5 mpirun -np 1 -mca pml ^yalla -mca mtl ^mxm -mca coll ^hcoll 
>>>> R
>>>> --slave < mk.R
>>>> 
>>>> Notice -np 1 - this is apparently how you start Rmpi jobs: ranks are 
>>>> spawned
>>>> by R dynamically inside the script

Re: [OMPI users] A hang in Rmpi at PMIx_Disconnect

[r...@open-mpi.org]

It might call disconnect more than once if it creates multiple communicators. 
Here’s another test case for that behavior:



intercomm_create.c
Description: Binary data



> On Jun 4, 2018, at 7:08 AM, Bennet Fauber  wrote:
> 
> Just out of curiosity, but would using Rmpi and/or doMPI help in any way?
> 
> -- bennet
> 
> 
> On Mon, Jun 4, 2018 at 10:00 AM, marcin.krotkiewski
>  wrote:
>> Thanks, Ralph!
>> 
>> Your code finishes normally, I guess then the reason might be lying in R.
>> Running the R code with -mca pmix_base_verbose 1 i see that each rank calls
>> ext2x:client disconnect twice (each PID prints the line twice)
>> 
>> [...]
>>3 slaves are spawned successfully. 0 failed.
>> [localhost.localdomain:11659] ext2x:client disconnect
>> [localhost.localdomain:11661] ext2x:client disconnect
>> [localhost.localdomain:11658] ext2x:client disconnect
>> [localhost.localdomain:11646] ext2x:client disconnect
>> [localhost.localdomain:11658] ext2x:client disconnect
>> [localhost.localdomain:11659] ext2x:client disconnect
>> [localhost.localdomain:11661] ext2x:client disconnect
>> [localhost.localdomain:11646] ext2x:client disconnect
>> 
>> In your example it's only called once per process.
>> 
>> Do you have any suspicion where the second call comes from? Might this be
>> the reason for the hang?
>> 
>> Thanks!
>> 
>> Marcin
>> 
>> 
>> On 06/04/2018 03:16 PM, r...@open-mpi.org wrote:
>> 
>> Try running the attached example dynamic code - if that works, then it
>> likely is something to do with how R operates.
>> 
>> 
>> 
>> 
>> 
>> On Jun 4, 2018, at 3:43 AM, marcin.krotkiewski
>>  wrote:
>> 
>> Hi,
>> 
>> I have some problems running R + Rmpi with OpenMPI 3.1.0 + PMIx 2.1.1. A
>> simple R script, which starts a few tasks, hangs at the end on diconnect.
>> Here is the script:
>> 
>> library(parallel)
>> numWorkers <- as.numeric(Sys.getenv("SLURM_NTASKS")) - 1
>> myCluster <- makeCluster(numWorkers, type = "MPI")
>> stopCluster(myCluster)
>> 
>> And here is how I run it:
>> 
>> SLURM_NTASKS=5 mpirun -np 1 -mca pml ^yalla -mca mtl ^mxm -mca coll ^hcoll R
>> --slave < mk.R
>> 
>> Notice -np 1 - this is apparently how you start Rmpi jobs: ranks are spawned
>> by R dynamically inside the script. So I ran into a number of issues here:
>> 
>> 1. with HPCX it seems that dynamic starting of ranks is not supported, hence
>> I had to turn off all of yalla/mxm/hcoll
>> 
>> --
>> Your application has invoked an MPI function that is not supported in
>> this environment.
>> 
>>  MPI function: MPI_Comm_spawn
>>  Reason:   the Yalla (MXM) PML does not support MPI dynamic process
>> functionality
>> --
>> 
>> 2. when I do that, the program does create a 'cluster' and starts the ranks,
>> but hangs in PMIx at MPI Disconnect. Here is the top of the trace from gdb:
>> 
>> #0  0x7f66b1e1e995 in pthread_cond_wait@@GLIBC_2.3.2 () from
>> /lib64/libpthread.so.0
>> #1  0x7f669eaeba5b in PMIx_Disconnect (procs=procs@entry=0x2e25d20,
>> nprocs=nprocs@entry=10, info=info@entry=0x0, ninfo=ninfo@entry=0) at
>> client/pmix_client_connect.c:232
>> #2  0x7f669ed6239c in ext2x_disconnect (procs=0x7ffd58322440) at
>> ext2x_client.c:1432
>> #3  0x7f66a13bc286 in ompi_dpm_disconnect (comm=0x2cc0810) at
>> dpm/dpm.c:596
>> #4  0x7f66a13e8668 in PMPI_Comm_disconnect (comm=0x2cbe058) at
>> pcomm_disconnect.c:67
>> #5  0x7f66a16799e9 in mpi_comm_disconnect () from
>> /cluster/software/R-packages/3.5/Rmpi/libs/Rmpi.so
>> #6  0x7f66b2563de5 in do_dotcall () from
>> /cluster/software/R/3.5.0/lib64/R/lib/libR.so
>> #7  0x7f66b25a207b in bcEval () from
>> /cluster/software/R/3.5.0/lib64/R/lib/libR.so
>> #8  0x7f66b25b0fd0 in Rf_eval.localalias.34 () from
>> /cluster/software/R/3.5.0/lib64/R/lib/libR.so
>> #9  0x7f66b25b2c62 in R_execClosure () from
>> /cluster/software/R/3.5.0/lib64/R/lib/libR.so
>> 
>> Might this also be related to the dynamic rank creation in R?
>> 
>> Thanks!
>> 
>> Marcin
>> 
>> ___
>> users mailing list
>> users@lists.open-mpi.org
>> https://lists.open-mpi.org/mailman/listinfo/users
>> 
>> 
>> 
>> ___
>> users mailing list
>> users@lists.open-mpi.org
>> https://lists.open-mpi.org/mailman/listinfo/users
>> 
>> 
>> 
>> ___
>> users mailing list
>> users@lists.open-mpi.org
>> https://lists.open-mpi.org/mailman/listinfo/users
> ___
> users mailing list
> users@lists.open-mpi.org
> https://lists.open-mpi.org/mailman/listinfo/users

___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users

Re: [OMPI users] A hang in Rmpi at PMIx_Disconnect

[r...@open-mpi.org]

Try running the attached example dynamic code - if that works, then it likely 
is something to do with how R operates.



simple_spawn.c
Description: Binary data



> On Jun 4, 2018, at 3:43 AM, marcin.krotkiewski  
> wrote:
> 
> Hi,
> 
> I have some problems running R + Rmpi with OpenMPI 3.1.0 + PMIx 2.1.1. A 
> simple R script, which starts a few tasks, hangs at the end on diconnect. 
> Here is the script:
> 
> library(parallel)
> numWorkers <- as.numeric(Sys.getenv("SLURM_NTASKS")) - 1
> myCluster <- makeCluster(numWorkers, type = "MPI")
> stopCluster(myCluster)
> 
> And here is how I run it:
> 
> SLURM_NTASKS=5 mpirun -np 1 -mca pml ^yalla -mca mtl ^mxm -mca coll ^hcoll R 
> --slave < mk.R
> 
> Notice -np 1 - this is apparently how you start Rmpi jobs: ranks are spawned 
> by R dynamically inside the script. So I ran into a number of issues here:
> 
> 1. with HPCX it seems that dynamic starting of ranks is not supported, hence 
> I had to turn off all of yalla/mxm/hcoll
> 
> --
> Your application has invoked an MPI function that is not supported in
> this environment.
> 
>   MPI function: MPI_Comm_spawn
>   Reason:   the Yalla (MXM) PML does not support MPI dynamic process 
> functionality
> --
> 
> 2. when I do that, the program does create a 'cluster' and starts the ranks, 
> but hangs in PMIx at MPI Disconnect. Here is the top of the trace from gdb:
> 
> #0  0x7f66b1e1e995 in pthread_cond_wait@@GLIBC_2.3.2 () from 
> /lib64/libpthread.so.0
> #1  0x7f669eaeba5b in PMIx_Disconnect (procs=procs@entry=0x2e25d20, 
> nprocs=nprocs@entry=10, info=info@entry=0x0, ninfo=ninfo@entry=0) at 
> client/pmix_client_connect.c:232
> #2  0x7f669ed6239c in ext2x_disconnect (procs=0x7ffd58322440) at 
> ext2x_client.c:1432
> #3  0x7f66a13bc286 in ompi_dpm_disconnect (comm=0x2cc0810) at 
> dpm/dpm.c:596
> #4  0x7f66a13e8668 in PMPI_Comm_disconnect (comm=0x2cbe058) at 
> pcomm_disconnect.c:67
> #5  0x7f66a16799e9 in mpi_comm_disconnect () from 
> /cluster/software/R-packages/3.5/Rmpi/libs/Rmpi.so
> #6  0x7f66b2563de5 in do_dotcall () from 
> /cluster/software/R/3.5.0/lib64/R/lib/libR.so
> #7  0x7f66b25a207b in bcEval () from 
> /cluster/software/R/3.5.0/lib64/R/lib/libR.so
> #8  0x7f66b25b0fd0 in Rf_eval.localalias.34 () from 
> /cluster/software/R/3.5.0/lib64/R/lib/libR.so
> #9  0x7f66b25b2c62 in R_execClosure () from 
> /cluster/software/R/3.5.0/lib64/R/lib/libR.so
> 
> Might this also be related to the dynamic rank creation in R?
> 
> Thanks!
> 
> Marcin
> 
> ___
> users mailing list
> users@lists.open-mpi.org
> https://lists.open-mpi.org/mailman/listinfo/users

___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users

Re: [OMPI users] slurm configuration override mpirun command line process mapping

[r...@open-mpi.org]

mpirun takes the #slots for each node from the slurm allocation. Your hostfile 
(at least, what you provided) retained that information and shows 2 slots on 
each node. So both the original allocation _and_ your constructed hostfile are 
both telling mpirun to assign 2 slots on each node.

Like I said before, on this old version, -H doesn’t say anything about #slots - 
that information is coming solely from the original allocation and your 
hostfile.


> On May 17, 2018, at 5:11 AM, Nicolas Deladerriere 
>  wrote:
> 
> About "-H" option and using --bynode option:
> 
> In my case, I do not specify number of slots by node to openmpi (see mpirun 
> command just above). From what I see the only place I define number of slots 
> in this case is actually through SLURM configuration 
> (SLURM_JOB_CPUS_PER_NODE=4(x3)). And I was not expected this to be taken when 
> running mpi processes.
> 
> Using --bynode is probably the easiest solution in my case, even if I am 
> scared that it will not necessary fit all my running configuration. Better 
> solution would be to review my management script for better integration with 
> slurm resources manager, but this is another story.

___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users

Re: [OMPI users] slurm configuration override mpirun command line process mapping

[r...@open-mpi.org]

The problem here is that you have made an incorrect assumption. In the older 
OMPI versions, the -H option simply indicated that the specified hosts were 
available for use - it did not imply the number of slots on that host. Since 
you have specified 2 slots on each host, and you told mpirun to launch 2 procs 
of your second app_context (the “slave”), it filled the first node with the 2 
procs.

I don’t recall the options for that old a version, but IIRC you should add 
--pernode to the cmd line to get exactly 1 proc/node

Or upgrade to a more recent OMPI version where -H can also be used to specify 
the #slots on a node :-)


> On May 15, 2018, at 11:58 PM, Gilles Gouaillardet 
>  wrote:
> 
> You can try to disable SLURM :
> 
> mpirun --mca ras ^slurm --mca plm ^slurm --mca ess ^slurm,slurmd ...
> 
> That will require you are able to SSH between compute nodes.
> Keep in mind this is far form ideal since it might leave some MPI
> processes on nodes if you cancel a job, and mess SLURM accounting too.
> 
> 
> Cheers,
> 
> Gilles
> 
> On Wed, May 16, 2018 at 3:50 PM, Nicolas Deladerriere
>  wrote:
>> Hi all,
>> 
>> 
>> 
>> I am trying to run mpi application through SLURM job scheduler. Here is my
>> running sequence
>> 
>> 
>> sbatch --> my_env_script.sh --> my_run_script.sh --> mpirun
>> 
>> 
>> In order to minimize modification of my production environment, I had to
>> setup following hostlist management in different scripts:
>> 
>> 
>> my_env_script.sh
>> 
>> 
>> build host list from SLURM resource manager information
>> 
>> Example: node01 nslots=2 ; node02 nslots=2 ; node03 nslots=2
>> 
>> 
>> my_run_script.sh
>> 
>> 
>> Build host list according to required job (process mapping depends on job
>> requirement).
>> 
>> Nodes are always fully dedicated to my job, but I have to manage different
>> master-slave situation with corresponding mpirun command:
>> 
>> as many process as number of slots:
>> 
>> mpirun -H node01 -np 1 process_master.x : -H node02,node02,node03,node03 -np
>> 4 process_slave.x
>> 
>> only one process per node (slots are usually used through openMP threading)
>> 
>> mpirun -H node01 -np 1 other_process_master.x : -H node02,node03 -np 2
>> other_process_slave.x
>> 
>> 
>> 
>> However, I realized that whatever I specified through my mpirun command,
>> process mapping is overridden at run time by slurm according to slurm
>> setting (either default setting or sbatch command line). For example, if I
>> run with:
>> 
>> 
>> sbatch -N 3 --exclusive my_env_script.sh myjob
>> 
>> 
>> where final mpirun command (depending on myjob) is:
>> 
>> 
>> mpirun -H node01 -np 1 other_process_master.x : -H node02,node03 -np 2
>> other_process_slave.x
>> 
>> 
>> It will be run with process mapping corresponding to:
>> 
>> 
>> mpirun -H node01 -np 1 other_process_master.x : -H node02,node02 -np 2
>> other_process_slave.x
>> 
>> 
>> So far I did not find a way to force mpirun to run with host mapping from
>> command line instead of slurm one. Is there a way to do it (either by using
>> MCA parameters of slurm configuration or …) ?
>> 
>> 
>> openmpi version : 1.6.5
>> 
>> slurm version : 17.11.2
>> 
>> 
>> 
>> Ragards,
>> 
>> Nicolas
>> 
>> 
>> Note 1: I know, it would be better to let slurm manage my process mapping by
>> only using slurm parameters and not specifying host mapping in my mpirun
>> command, but in order to minimize modification in my production environment
>> I had to use such solution.
>> 
>> Note 2: I know I am using old openmpi version !
>> 
>> 
>> ___
>> users mailing list
>> users@lists.open-mpi.org
>> https://lists.open-mpi.org/mailman/listinfo/users
> ___
> users mailing list
> users@lists.open-mpi.org
> https://lists.open-mpi.org/mailman/listinfo/users

___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users

Re: [OMPI users] OpenMPI 3.0.1 - mpirun hangs with 2 hosts

[r...@open-mpi.org]

You got that error because the orted is looking for its rank on the cmd line 
and not finding it.


> On May 14, 2018, at 12:37 PM, Max Mellette  wrote:
> 
> Hi Gus,
> 
> Thanks for the suggestions. The correct version of openmpi seems to be 
> getting picked up; I also prepended .bashrc with the installation path like 
> you suggested, but it didn't seemed to help:
> 
> user@b09-30:~$ cat .bashrc
> export 
> PATH=/home/user/openmpi_install/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
> export LD_LIBRARY_PATH=/home/user/openmpi_install/lib
> user@b09-30:~$ which mpicc
> /home/user/openmpi_install/bin/mpicc
> user@b09-30:~$ /usr/bin/ssh -x b09-32 orted
> [b09-32:204536] [[INVALID],INVALID] ORTE_ERROR_LOG: Not found in file 
> ess_env_module.c at line 147
> [b09-32:204536] [[INVALID],INVALID] ORTE_ERROR_LOG: Bad parameter in file 
> util/session_dir.c at line 106
> [b09-32:204536] [[INVALID],INVALID] ORTE_ERROR_LOG: Bad parameter in file 
> util/session_dir.c at line 345
> [b09-32:204536] [[INVALID],INVALID] ORTE_ERROR_LOG: Bad parameter in file 
> base/ess_base_std_orted.c at line 270
> --
> It looks like orte_init failed for some reason; your parallel process is
> likely to abort.  There are many reasons that a parallel process can
> fail during orte_init; some of which are due to configuration or
> environment problems.  This failure appears to be an internal failure;
> here's some additional information (which may only be relevant to an
> Open MPI developer):
> 
>   orte_session_dir failed
>   --> Returned value Bad parameter (-5) instead of ORTE_SUCCESS
> --
> 
> Thanks,
> Max
> 
> 
> On Mon, May 14, 2018 at 11:41 AM, Gus Correa  > wrote:
> Hi Max
> 
> Just in case, as environment mix often happens.
> Could it be that you are inadvertently picking another
> installation of OpenMPI, perhaps installed from packages
> in /usr , or /usr/local?
> That's easy to check with 'which mpiexec' or
> 'which mpicc', for instance.
> 
> Have you tried to prepend (as opposed to append) OpenMPI
> to your PATH? Say:
> 
> export 
> PATH='/home/user/openmpi_install/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin'
> 
> I hope this helps,
> Gus Correa
> 
> ___
> users mailing list
> users@lists.open-mpi.org
> https://lists.open-mpi.org/mailman/listinfo/users

___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users

Re: [OMPI users] Openmpi-3.1.0 + slurm (fixed)

[r...@open-mpi.org]

Good news - thanks!

> On May 8, 2018, at 7:02 PM, Bill Broadley  wrote:
> 
> 
> Sorry all,
> 
> Chris S over on the slurm list spotted it right away.  I didn't have the
> MpiDefault set to pmix_v2.
> 
> I can confirm that Ubuntu 18.04, gcc-7.3, openmpi-3.1.0, pmix-2.1.1, and
> slurm-17.11.5 seem to work well together.
> 
> Sorry for the bother.
> 
> ___
> users mailing list
> users@lists.open-mpi.org
> https://lists.open-mpi.org/mailman/listinfo/users

___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users

Re: [OMPI users] Memory leak with pmix_finalize not being called

[r...@open-mpi.org]

Ouch - trivial fix. I’m inserting it into PMIx and will provide it to the OMPI 
team

> On May 4, 2018, at 5:20 AM, Saurabh T  wrote:
> 
> This is with valgrind 3.0.1 on a Centos 6 system. It appears pmix_finalize 
> isnt called and this reports leaks from valgrind despite the provided 
> suppression file being used. A cursory check reveals MPI_Finalize calls 
> pmix_rte_finalize which decrements pmix_initialized to 0 before calling 
> pmix_cleanup. pmix_cleanup sees the variable is 0 and does not call 
> pmix_finalize. See 
> https://github.com/pmix/pmix/blob/master/src/runtime/pmix_finalize.c.  Is 
> this a bug or something I am doing wrong? Thank you.
> 
> ___
> users mailing list
> users@lists.open-mpi.org
> https://lists.open-mpi.org/mailman/listinfo/users

___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users

Re: [OMPI users] openmpi/slurm/pmix

[r...@open-mpi.org]

> On Apr 25, 2018, at 8:16 AM, Michael Di Domenico <mdidomeni...@gmail.com> 
> wrote:
> 
> On Mon, Apr 23, 2018 at 6:07 PM, r...@open-mpi.org <r...@open-mpi.org> wrote:
>> Looks like the problem is that you didn’t wind up with the external PMIx. 
>> The component listed in your error is the internal PMIx one which shouldn’t 
>> have built given that configure line.
>> 
>> Check your config.out and see what happened. Also, ensure that your 
>> LD_LIBRARY_PATH is properly pointing to the installation, and that you built 
>> into a “clean” prefix.
> 
> the "clean prefix" part seemed to fix my issue.  i'm not exactly sure
> i understand why/how though.  i recompiled pmix and removed the old
> installation before doing a make install

When you build, we don’t automatically purge the prefix location of any prior 
libraries. Thus, the old install of the internal PMIx library was still 
present. It has a higher priority than the external components, and so it was 
being picked up and used.

Starting clean removed it, leaving the external component to be selected.

> 
> when i recompiled openmpi it seems to have figured itself out
> 
> i think things are still a little wonky, but at least that issue is gone
> ___
> users mailing list
> users@lists.open-mpi.org
> https://lists.open-mpi.org/mailman/listinfo/users

___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users

Re: [OMPI users] Error in hello_cxx.cc

[r...@open-mpi.org]

and 3) we no longer support Windows. You could try using the cygwin port 
instead.

> On Apr 23, 2018, at 7:52 PM, Nathan Hjelm  wrote:
> 
> Two things. 1) 1.4 is extremely old and you will not likely get much help 
> with it, and 2) the c++ bindings were deprecated in MPI-2.2 (2009) and 
> removed in MPI-3.0 (2012) so you probably want to use the C bindings instead.
> 
> -Nathan
> 
> On Apr 23, 2018, at 8:14 PM, Amir via users  > wrote:
> 
>> Yes, I am running under windows using visual studio 2010 express edition. 
>> The build process is done fine but when I am trying to run I would get the 
>> error code 6 in MPI::Init().  I have installed openmpi-1.4.5 . I have also 
>> attached the log file of the CMake hope this would help. The screenshot of 
>> ipconfig is also attached. 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> ___
>> users mailing list
>> users@lists.open-mpi.org 
>> https://lists.open-mpi.org/mailman/listinfo/users 
>> ___
> users mailing list
> users@lists.open-mpi.org
> https://lists.open-mpi.org/mailman/listinfo/users

___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users

Re: [OMPI users] openmpi/slurm/pmix

[r...@open-mpi.org]

Hi Michael

Looks like the problem is that you didn’t wind up with the external PMIx. The 
component listed in your error is the internal PMIx one which shouldn’t have 
built given that configure line.

Check your config.out and see what happened. Also, ensure that your 
LD_LIBRARY_PATH is properly pointing to the installation, and that you built 
into a “clean” prefix.


> On Apr 23, 2018, at 12:01 PM, Michael Di Domenico  
> wrote:
> 
> i'm trying to get slurm 17.11.5 and openmpi 3.0.1 working with pmix.
> 
> everything compiled, but when i run something it get
> 
> : symbol lookup error: /openmpi/mca_pmix_pmix2x.so: undefined symbol:
> opal_libevent2022_evthread_use_pthreads
> 
> i more then sure i did something wrong, but i'm not sure what, here's what i 
> did
> 
> compile libevent 2.1.8
> 
> ./configure --prefix=/libevent-2.1.8
> 
> compile pmix 2.1.0
> 
> ./configure --prefix=/pmix-2.1.0 --with-psm2
> --with-munge=/munge-0.5.13 --with-libevent=/libevent-2.1.8
> 
> compile openmpi
> 
> ./configure --prefix=/openmpi-3.0.1 --with-slurm=/slurm-17.11.5
> --with-hwloc=external --with-mxm=/opt/mellanox/mxm
> --with-cuda=/usr/local/cuda --with-pmix=/pmix-2.1.0
> --with-libevent=/libevent-2.1.8
> 
> when i look at the symbols in the mca_pmix_pmix2x.so library the
> function is indeed undefined (U) in the output, but checking ldd
> against the library doesn't show any missing
> 
> any thoughts?
> ___
> users mailing list
> users@lists.open-mpi.org
> https://lists.open-mpi.org/mailman/listinfo/users

___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users

Re: [OMPI users] Error in hello_cxx.cc

[r...@open-mpi.org]

Also, I note from the screenshot that you appear to be running on Windows with 
a Windows binary. Correct?


> On Apr 23, 2018, at 7:08 AM, Jeff Squyres (jsquyres)  
> wrote:
> 
> Can you send all the information listed here:
> 
>https://www.open-mpi.org/community/help/
> 
> 
> 
>> On Apr 22, 2018, at 2:28 PM, Amir via users  wrote:
>> 
>> Hi everybody,
>> 
>> After having some problems with setting up the debugging environment for 
>> Visual Studio 10, I am trying to debug the first Open_MPI example program 
>> (hello_cxx.cc) . 
>> 
>> I am getting an error in the function call MPI::Init();  . The attached 
>> screenshot should clarify this better.  I guess this is related to the rank 
>> but don't have any idea why and how to fix it. 
>> 
>> Any guidance is highly appreciated.
>> 
>> Thanks you,
>> 
>> Amir
>> ___
>> users mailing list
>> users@lists.open-mpi.org
>> https://lists.open-mpi.org/mailman/listinfo/users
> 
> 
> -- 
> Jeff Squyres
> jsquy...@cisco.com
> 
> ___
> users mailing list
> users@lists.open-mpi.org
> https://lists.open-mpi.org/mailman/listinfo/users

___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users

Re: [OMPI users] ARM/Allinea DDT

[r...@open-mpi.org]

This inadvertently was sent directly to me, and Ryan asked if I could please 
post it on his behalf - he needs to get fully approved on the mailing list.

Ralph


> On Apr 11, 2018, at 1:19 PM, Ryan Hulguin  wrote:
> 
> Hello Charlie Taylor,
>  
> I have replied to your support ticket indicating that we do not claim to 
> fully support OpenMPI 3.0, since we are still running tests ourselves to 
> ensure everything works smoothly with DDT.
> However, we will work with you to help identify the problems you are 
> encountering, and to determine whether or not anything can be done at this 
> time.
> Please provide more detailed information in the support ticket you have open 
> with us so that we can be of further assistance.
>  
> Regards,
> Ryan Hulguin
>  
> IMPORTANT NOTICE: The contents of this email and any attachments are 
> confidential and may also be privileged. If you are not the intended 
> recipient, please notify the sender immediately and do not disclose the 
> contents to any other person, use it for any purpose, or store or copy the 
> information in any medium. Thank you.

___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users

Re: [OMPI users] ARM/Allinea DDT

[r...@open-mpi.org]

You probably should provide a little more info here. I know the MPIR attach was 
broken in the v2.x series, but we fixed that - could be something remains 
broken in OMPI 3.x.

FWIW: I doubt it's an Allinea problem.

> On Apr 11, 2018, at 11:54 AM, Charles A Taylor  wrote:
> 
> 
> Contacting ARM seems a bit difficult so I thought I would ask here.  We rely 
> on DDT for debugging but it doesn’t work with OpenMPI 3.x and I can’t find 
> anything about them having plans to support it.
> 
> Anyone know if ARM DDT has plans to support newer versions of OpenMPI?
> 
> Charlie Taylor
> UF Research Computing
> ___
> users mailing list
> users@lists.open-mpi.org
> https://lists.open-mpi.org/mailman/listinfo/users

___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users

Re: [OMPI users] OpenMPI 3.0.0 on RHEL-7

[r...@open-mpi.org]

So long as it is libevent-devel-2.0.22, you should be okay. You might want to 
up PMIx to v1.2.5 as Slurm 16.05 should handle that okay. OMPI v3.0.0 has PMIx 
2.0 in it, but should be okay with 1.2.5 last I checked (but it has been awhile 
and I can’t swear to it).


> On Mar 7, 2018, at 2:03 PM, Charles A Taylor  wrote:
> 
> Hi 
> 
> Distro: RHEL-7 (7.4)
> SLURM: 16.05.11
> PMIx: 1.1.5
> 
> Trying to build OpenMPI 3.0.0 for our RHEL7 systems but running into what 
> might be a configure script issue more than a real incompatibility problem.  
> Configuring with the following,
> 
>   --with-slurm=/opt/slurm --with-pmix=/opt/pmix 
> --with-external-libpmix=/opt/pmix/lib64 --with-libevent=/usr 
> 
> It seems happy enough with PMIx and there appears to be PMIx 1.x support.  
> However, libevent is another issue and my take so far is that the EL7 native 
> libevent-devel-2.0.x is the crux of the problem.  It doesn’t seem like the 
> configure script expects that and I’m a little reluctant to start patching 
> things together with symlinks or configure script changes.
> 
> My real question is what is intended?   Should this be working or was 3.0.0 
> really intended for PMIx 2.x?  I imagine we will go to PMIx 2.x with SLURM 17 
> but aren’t quite ready for that yet.
> 
> Regards,
> 
> Charlie Taylor
> UF Research Computing
> 
> 
> ___
> users mailing list
> users@lists.open-mpi.org
> https://lists.open-mpi.org/mailman/listinfo/users

___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users

Re: [OMPI users] libopen-pal not found

[r...@open-mpi.org]

Not that I’ve heard - you need to put it in your LD_LIBRARY_PATH


> On Mar 2, 2018, at 10:15 AM, Mahmood Naderan  wrote:
> 
> Hi,
> After a successful installation of opmi v3 with cuda enabled, I see that ldd 
> can not find a right lib file although it exists. /usr/local/lib is one of 
> the default locations for the library files. Isn't that?
> 
> 
> 
> $ which mpic++
> /usr/local/bin/mpic++
> $ ldd /usr/local/bin/mpic++
> linux-vdso.so.1 =>  (0x7fff06d0d000)
> libopen-pal.so.40 => not found
> libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 
> (0x7f3901cbd000)
> libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x7f39018f2000)
> /lib64/ld-linux-x86-64.so.2 (0x55df58104000)
> $ sudo find /usr/local -name libopen-pal*
> /usr/local/lib/libopen-pal.so.40
> /usr/local/lib/libopen-pal.so.40.0.0
> /usr/local/lib/libopen-pal.la 
> /usr/local/lib/libopen-pal.so
> 
> 
> 
> Regards,
> Mahmood
> 
> 
> ___
> users mailing list
> users@lists.open-mpi.org
> https://lists.open-mpi.org/mailman/listinfo/users

___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users

Re: [OMPI users] Cannot compile 1.10.2 under CentOS 7 rdma-core-devel-13-7.el7.x86_64

[r...@open-mpi.org]

Not unless you have a USNIC card in your machine!


> On Feb 28, 2018, at 8:08 AM, William T Jones <w.t.jo...@nasa.gov> wrote:
> 
> Thank you!
> 
> Will that have any adverse side effects?
> Performance penalties?
> 
> On 02/28/2018 10:57 AM, r...@open-mpi.org wrote:
>> Add --without-usnic
>>> On Feb 28, 2018, at 7:50 AM, William T Jones <w.t.jo...@nasa.gov> wrote:
>>> 
>>> I realize that OpenMPI 1.10.2 is quite old, however, for compatibility I
>>> am attempting to compile it after a system upgrade to CentOS 7.
>>> 
>>> This system does include infiniband and I have configured as follows
>>> using Intel 2017.2.174 compilers:
>>> 
>>> % ./configure --enable-static \
>>>  --with-tm=/usr/local/pkgs/PBSPro_64 \
>>>  --enable-mpi-thread-multiple \
>>>  --with-verbs=/usr \
>>>  --enable-mpi-cxx \
>>>  FC=ifort \
>>>  F77=ifort \
>>>  CC=icc \
>>>  CXX=icpc \
>>>  CFLAGS="-O3 -ip" \
>>>  FCFLAGS="-O3 -ip" \
>>>  LIBS=-lcrypto -lpthread
>>> 
>>> However, when I compile I get the following error:
>>> 
>>>  ...
>>>  Making all in mca/common/verbs_usnic
>>>  make[2]: Entering directory
>>> `/usr/src/openmpi-1.10.2/ompi/mca/common/verbs_usnic'
>>>CC   libmca_common_verbs_usnic_la-common_verbs_usnic_fake.lo
>>>  common_verbs_usnic_fake.c(72): error: struct "ibv_device" has no field
>>> "ops"
>>>.ops = {
>>> ^
>>> 
>>>  common_verbs_usnic_fake.c(89): warning #266: function
>>> "ibv_read_sysfs_file" declared implicitly
>>>if (ibv_read_sysfs_file(uverbs_sys_path, "device/vendor",
>>>^
>>> 
>>>  common_verbs_usnic_fake.c(133): warning #266: function
>>> "ibv_register_driver" declared implicitly
>>>ibv_register_driver("usnic_verbs", fake_driver_init);
>>>^
>>> 
>>>  compilation aborted for common_verbs_usnic_fake.c (code 2)
>>> 
>>> 
>>> Unfortunately, my /usr/include/infiniband/verbs.h file defines the
>>> "ibv_device" structure but does not include "ops" member.  Instead the
>>> structure is defined as follows:
>>> 
>>>  /* Obsolete, never used, do not touch */
>>>  struct _ibv_device_ops {
>>>  struct ibv_context *(*_dummy1)(struct ibv_device *device,
>>> int cmd_fd);
>>>  void(*_dummy2)(struct ibv_context *context);
>>>  };
>>> 
>>>  enum {
>>>  IBV_SYSFS_NAME_MAX  = 64,
>>>  IBV_SYSFS_PATH_MAX  = 256
>>>  };
>>> 
>>>  struct ibv_device {
>>>  struct _ibv_device_ops  _ops;
>>>  enum ibv_node_type  node_type;
>>>  enum ibv_transport_type transport_type;
>>>  /* Name of underlying kernel IB device, eg "mthca0" */
>>>  charname[IBV_SYSFS_NAME_MAX];
>>>  /* Name of uverbs device, eg "uverbs0" */
>>>  chardev_name[IBV_SYSFS_NAME_MAX];
>>>  /* Path to infiniband_verbs class device in sysfs */
>>>  chardev_path[IBV_SYSFS_PATH_MAX];
>>>  /* Path to infiniband class device in sysfs */
>>>  charibdev_path[IBV_SYSFS_PATH_MAX];
>>>  };
>>> 
>>> 
>>> OpenMPI was previously compiled successfully under CentOS 6 and every
>>> indication is that the /usr/include/infiniband/verbs.h was defined
>>> similarly (again without the "ops" member).
>>> 
>>> Is it possible that there is a configure option that prevents this source 
>>> from being included in the build?
>>> 
>>> Any help is appreciated,
>>> 
>>> 
>>> -- 
>>> =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
>>> 
>>>Bill Jones   w.t.jo...@nasa.gov
>>>Mail Stop 128 Computational AeroSciences Branch
>>>15 Langley Boulevard   Research Directorate
>>>NASA Langley Research Center  

Re: [OMPI users] Cannot compile 1.10.2 under CentOS 7 rdma-core-devel-13-7.el7.x86_64

[r...@open-mpi.org]

Add --without-usnic


> On Feb 28, 2018, at 7:50 AM, William T Jones  wrote:
> 
> I realize that OpenMPI 1.10.2 is quite old, however, for compatibility I
> am attempting to compile it after a system upgrade to CentOS 7.
> 
> This system does include infiniband and I have configured as follows
> using Intel 2017.2.174 compilers:
> 
> % ./configure --enable-static \
>  --with-tm=/usr/local/pkgs/PBSPro_64 \
>  --enable-mpi-thread-multiple \
>  --with-verbs=/usr \
>  --enable-mpi-cxx \
>  FC=ifort \
>  F77=ifort \
>  CC=icc \
>  CXX=icpc \
>  CFLAGS="-O3 -ip" \
>  FCFLAGS="-O3 -ip" \
>  LIBS=-lcrypto -lpthread
> 
> However, when I compile I get the following error:
> 
>  ...
>  Making all in mca/common/verbs_usnic
>  make[2]: Entering directory
> `/usr/src/openmpi-1.10.2/ompi/mca/common/verbs_usnic'
>CC   libmca_common_verbs_usnic_la-common_verbs_usnic_fake.lo
>  common_verbs_usnic_fake.c(72): error: struct "ibv_device" has no field
> "ops"
>.ops = {
> ^
> 
>  common_verbs_usnic_fake.c(89): warning #266: function
> "ibv_read_sysfs_file" declared implicitly
>if (ibv_read_sysfs_file(uverbs_sys_path, "device/vendor",
>^
> 
>  common_verbs_usnic_fake.c(133): warning #266: function
> "ibv_register_driver" declared implicitly
>ibv_register_driver("usnic_verbs", fake_driver_init);
>^
> 
>  compilation aborted for common_verbs_usnic_fake.c (code 2)
> 
> 
> Unfortunately, my /usr/include/infiniband/verbs.h file defines the
> "ibv_device" structure but does not include "ops" member.  Instead the
> structure is defined as follows:
> 
>  /* Obsolete, never used, do not touch */
>  struct _ibv_device_ops {
>  struct ibv_context *(*_dummy1)(struct ibv_device *device,
> int cmd_fd);
>  void(*_dummy2)(struct ibv_context *context);
>  };
> 
>  enum {
>  IBV_SYSFS_NAME_MAX  = 64,
>  IBV_SYSFS_PATH_MAX  = 256
>  };
> 
>  struct ibv_device {
>  struct _ibv_device_ops  _ops;
>  enum ibv_node_type  node_type;
>  enum ibv_transport_type transport_type;
>  /* Name of underlying kernel IB device, eg "mthca0" */
>  charname[IBV_SYSFS_NAME_MAX];
>  /* Name of uverbs device, eg "uverbs0" */
>  chardev_name[IBV_SYSFS_NAME_MAX];
>  /* Path to infiniband_verbs class device in sysfs */
>  chardev_path[IBV_SYSFS_PATH_MAX];
>  /* Path to infiniband class device in sysfs */
>  charibdev_path[IBV_SYSFS_PATH_MAX];
>  };
> 
> 
> OpenMPI was previously compiled successfully under CentOS 6 and every
> indication is that the /usr/include/infiniband/verbs.h was defined
> similarly (again without the "ops" member).
> 
> Is it possible that there is a configure option that prevents this source 
> from being included in the build?
> 
> Any help is appreciated,
> 
> 
> -- 
> =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
> 
>Bill Jones   w.t.jo...@nasa.gov
>Mail Stop 128 Computational AeroSciences Branch
>15 Langley Boulevard   Research Directorate
>NASA Langley Research Center   Building 1268, Room 1044
>Hampton, VA  23681-2199   Phone +1 757 864-5318
>Fax +1 757 864-8816
> http://fun3d.larc.nasa.gov
> ___
> users mailing list
> users@lists.open-mpi.org
> https://lists.open-mpi.org/mailman/listinfo/users

___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users

Re: [OMPI users] Using OMPI Standalone in a Windows/Cygwin Environment

[r...@open-mpi.org]

There are a couple of problems here. First the “^tcp,self,sm” is telling OMPI 
to turn off all three of those transports, which probably leaves you with 
nothing. What you really want is to restrict to shared memory, so your param 
should be “-mca btl self,sm”. This will disable all transports other than 
shared memory - note that you always must enable the “self” btl.

Second, you likely also need to ensure that the OOB isn’t trying to use tcp, so 
add “-mca oob ^tcp” to your cmd line. It shouldn’t be active anyway, but better 
safe.


> On Feb 26, 2018, at 9:14 AM, Michael A. Saverino 
>  wrote:
> 
> 
> I am running the v-1.10.7 OMPI package that is available via the Cygwin
> package manager.  I have a requirement to run my OMPI application
> standalone on a Windows/Cygwin system without any network connectivity. 
> If my OMPI system is not connected to the network, I get the following
> errors when I try to run my OMPI application:
>  
> [SAXM4WIN:02124] [[20996,1],0] tcp_peer_send_blocking: send() to socket
> 12 failed: Transport endpoint is not connected (128)
> [SAXM4WIN:02124] [[20996,1],0] tcp_peer_send_blocking: send() to socket
> 12 failed: Transport endpoint is not connected (128)
> 
> I have tried the following qualifiers in my OMPI command to no avail:
> 
> --mca btl ^tcp,self,sm
> 
> So the question is, am I able to disable TCP networking, either via
> command line or code, if I only plan to use cores on a single machine
> for OMPI execution?
> 
> Many Thanks,
> 
> Mike...
> 
> 
> 
> -- 
> Michael A.Saverino
> Contractor 
> Senior Engineer, Information Technology Division
> Code 5522
> Naval Research Laboratory 
> W (202)767-5652
> C (814)242-0217
> https://www.nrl.navy.mil/itd/ncs/
> 
> 
> ___
> users mailing list
> users@lists.open-mpi.org
> https://lists.open-mpi.org/mailman/listinfo/users

___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users

[OMPI users] Fabric manager interactions: request for comments

[r...@open-mpi.org]

Hello all

The PMIx community is starting work on the next phase of defining support for 
network interactions, looking specifically at things we might want to obtain 
and/or control via the fabric manager. A very preliminary draft is shown here:

https://pmix.org/home/pmix-standard/fabric-manager-roles-and-expectations/ 


We would welcome any comments/suggestions regarding information you might find 
useful to get regarding the network, or controls you would like to set.

Thanks in advance
Ralph

___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users

Re: [OMPI users] Setting mpirun default parameters in a file

[r...@open-mpi.org]

Set the MCA param “rmaps_base_oversubscribe=1” in your default MCA param file, 
or in your environment

> On Jan 10, 2018, at 4:42 AM, Florian Lindner  wrote:
> 
> Hello,
> 
> a recent openmpi update on my Arch machine seems to have enabled 
> --nooversubscribe, as described in the manpage. Since I
> regularly test on my laptop with just 2 physical cores, I want to set 
> --oversubscribe by default.
> 
> How can I do that?
> 
> I am also a bit surprised, that openmpi takes the physical number of cores 
> into account, not the logical (which is 4 on
> my machine).
> 
> Thanks,
> Florian
> ___
> users mailing list
> users@lists.open-mpi.org
> https://lists.open-mpi.org/mailman/listinfo/users

___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users

Re: [OMPI users] latest Intel CPU bug

[r...@open-mpi.org]

That is enough, folks. This is an email forum for users to get help regarding 
Open MPI, not a place to vent your feelings about specific vendors. We ask that 
you respect that policy and refrain from engaging in such behavior.

We don’t care if you are quoting someone else - the fact that “Mikey said it” 
doesn’t justify violating the policy. So please stop this here and now.

Thank you
Ralph


> On Jan 5, 2018, at 6:09 AM, John Chludzinski <john.chludzin...@gmail.com> 
> wrote:
> 
> I believe this snippet sums it up pretty well:
> 
> "Now you have a bit more context about why Intel’s response was, well, a 
> non-response. They blamed others, correctly, for having the same problem but 
> their blanket statement avoided the obvious issue of the others aren’t 
> crippled by the effects of the patches like Intel. Intel screwed up, badly, 
> and are facing a 30% performance hit going forward for it. AMD did right and 
> are probably breaking out the champagne at HQ about now."
> 
> On Fri, Jan 5, 2018 at 5:38 AM, Matthieu Brucher <matthieu.bruc...@gmail.com 
> <mailto:matthieu.bruc...@gmail.com>> wrote:
> Hi,
> 
> I think, on the contrary, that he did notice the AMD/ARM issue. I suppose you 
> haven't read the text (and I like the fact that there are different opinions 
> on this issue).
> 
> Matthieu
> 
> 2018-01-05 8:23 GMT+01:00 Gilles Gouaillardet <gil...@rist.or.jp 
> <mailto:gil...@rist.or.jp>>:
> John,
> 
> 
> The technical assessment so to speak is linked in the article and is 
> available at 
> https://googleprojectzero.blogspot.jp/2018/01/reading-privileged-memory-with-side.html
>  
> <https://googleprojectzero.blogspot.jp/2018/01/reading-privileged-memory-with-side.html>.
> 
> The long rant against Intel PR blinded you and you did not notice AMD and ARM 
> (and though not mentionned here, Power and Sparc too) are vulnerable to some 
> bugs.
> 
> 
> Full disclosure, i have no affiliation with Intel, but i am getting pissed 
> with the hysteria around this issue.
> 
> Gilles
> 
> 
> On 1/5/2018 3:54 PM, John Chludzinski wrote:
> That article gives the best technical assessment I've seen of Intel's 
> architecture bug. I noted the discussion's subject and thought I'd add some 
> clarity. Nothing more.
> 
> For the TL;DR crowd: get an AMD chip in your computer.
> 
> On Thursday, January 4, 2018, r...@open-mpi.org <mailto:r...@open-mpi.org> 
> <mailto:r...@open-mpi.org <mailto:r...@open-mpi.org>> <r...@open-mpi.org 
> <mailto:r...@open-mpi.org> <mailto:r...@open-mpi.org 
> <mailto:r...@open-mpi.org>>> wrote:
> 
> Yes, please - that was totally inappropriate for this mailing list.
> Ralph
> 
> 
> On Jan 4, 2018, at 4:33 PM, Jeff Hammond <jeff.scie...@gmail.com 
> <mailto:jeff.scie...@gmail.com>
> <mailto:jeff.scie...@gmail.com <mailto:jeff.scie...@gmail.com>>> wrote:
> 
> Can we restrain ourselves to talk about Open-MPI or at least
> technical aspects of HPC communication on this list and leave the
> stock market tips for Hacker News and Twitter?
> 
> Thanks,
> 
> Jeff
> 
> On Thu, Jan 4, 2018 at 3:53 PM, John
> Chludzinski<john.chludzin...@gmail.com <mailto:john.chludzin...@gmail.com>
> <mailto:john.chludzin...@gmail.com 
> <mailto:john.chludzin...@gmail.com>>>wrote:
> 
> 
> Fromhttps://semiaccurate.com/2018/01/04/kaiser-security-holes-will-devastate-intels-marketshare/
>  
> <http://semiaccurate.com/2018/01/04/kaiser-security-holes-will-devastate-intels-marketshare/>
> 
> <https://semiaccurate.com/2018/01/04/kaiser-security-holes-will-devastate-intels-marketshare/
>  
> <https://semiaccurate.com/2018/01/04/kaiser-security-holes-will-devastate-intels-marketshare/>>
> 
> 
> 
>   Kaiser security holes will devastate Intel’s marketshare
> 
> 
>   Analysis: This one tips the balance toward AMD in a big way
> 
> 
>   Jan 4, 2018 by Charlie Demerjian
>   <https://semiaccurate.com/author/charlie/ 
> <https://semiaccurate.com/author/charlie/>>
> 
> 
> 
> This latest decade-long critical security hole in Intel CPUs
> is going to cost the company significant market share.
> SemiAccurate thinks it is not only consequential but will
> shift the balance of power away from Intel CPUs for at least
> the next several years.
> 
> Today’s latest crop of gaping security flaws have three sets
> of holes across Intel, AMD, and ARM processors along with a
>   

Re: [OMPI users] latest Intel CPU bug

[r...@open-mpi.org]

> the bit about, “Zero AMD vulnerability or risk because of AMD architecture 
> differences.” See an issue here?
> 
> What AMD didn’t spell out in detail is a minor difference in 
> microarchitecture between Intel and AMD CPUs. When a CPU speculatively 
> executes and crosses a privilege level boundary, any idiot would probably say 
> that the CPU should see this crossing and not execute the following 
> instructions that are out of it’s privilege level. This isn’t rocket science, 
> just basic common sense.
> 
> AMD’s microarchitecture sees this privilege level change and throws the 
> microarchitectural equivalent of a hissy fit and doesn’t execute the code. 
> Common sense wins out. Intel’s implementation does execute the following code 
> across privilege levels which sounds on the surface like a bit of a face-palm 
> implementation but it really isn’t.
> 
> What saves Intel is that the speculative execution goes on but, to the best 
> of our knowledge, is unwound when the privilege level changes a few 
> instructions later. Since Intel CPUs in the wild don’t crash or violate 
> privilege levels, it looks like that mechanism works properly in practice. 
> What these new exploits do is slip a few very short instructions in that can 
> read data from the other user or privilege level before the context change 
> happens. If crafted correctly the instructions are unwound but the data can 
> be stashed in a place that is persistent.
> 
> Intel probably get a slight performance gain from doing this ‘sloppy’ method 
> but AMD seems to have have done the right thing for the right reasons. That 
> extra bounds check probably take a bit of time but in retrospect, doing the 
> right thing was worth it. Since both are fundamental ‘correct’ behaviors for 
> their respective microarchitectures, there is no possible fix, just code that 
> avoids scenarios where it can be abused.
> 
> For Intel this avoidance comes with a 30% performance hit on server type 
> workloads, less on desktop workloads. For AMD the problem was avoided by 
> design and the performance hit is zero. Doing the right thing for the right 
> reasons even if it is marginally slower seems to have paid off in this 
> circumstance. Mother was right, AMD listened, Intel didn’t.
> 
> Weasel Words:
> 
> Now you have a bit more context about why Intel’s response was, well, a 
> non-response. They blamed others, correctly, for having the same problem but 
> their blanket statement avoided the obvious issue of the others aren’t 
> crippled by the effects of the patches like Intel. Intel screwed up, badly, 
> and are facing a 30% performance hit going forward for it. AMD did right and 
> are probably breaking out the champagne at HQ about now.
> 
> Intel also tried to deflect lawyers by saying they follow industry best 
> practices. They don’t and the AMT hole was a shining example of them putting 
> PR above customer security. Similarly their sitting on the fix for the TXT 
> flaw for *THREE*YEARS* 
> <https://www.semiaccurate.com/2016/01/20/intel-puts-out-secure-cpus-based-on-insecurity/>
>  because they didn’t want to admit to architectural security blunders and 
> reveal publicly embarrassing policies until forced to disclose by a 
> governmental agency being exploited by a foreign power is another example 
> that shines a harsh light on their ‘best practices’ line. There are many more 
> like this. Intel isn’t to be trusted for security practices or disclosures 
> because PR takes precedence over customer security.
> 
> Rubber Meet Road:
> 
> Unfortunately security doesn’t sell and rarely affects marketshare. This time 
> however is different and will hit Intel were it hurts, in the wallet. 
> SemiAccurate thinks this exploit is going to devastate Intel’s marketshare. 
> Why? Read on subscribers.
> 
> Note: The following is analysis for professional level subscribers only.
> 
> Disclosures: Charlie Demerjian and Stone Arch Networking Services, Inc. have 
> no consulting relationships, investment relationships, or hold any investment 
> positions with any of the companies mentioned in this report.
> 
> 
> 
> On Thu, Jan 4, 2018 at 6:21 PM, Reuti <re...@staff.uni-marburg.de 
> <mailto:re...@staff.uni-marburg.de>> wrote:
> 
> Am 04.01.2018 um 23:45 schrieb r...@open-mpi.org <mailto:r...@open-mpi.org>:
> 
> > As more information continues to surface, it is clear that this original 
> > article that spurred this thread was somewhat incomplete - probably 
> > released a little too quickly, before full information was available. There 
> > is still some confusion out there, but the gist from surfing the various 
> > articles (and trimming away the hysteria) appears to be:

Re: [OMPI users] latest Intel CPU bug

[r...@open-mpi.org]

As more information continues to surface, it is clear that this original 
article that spurred this thread was somewhat incomplete - probably released a 
little too quickly, before full information was available. There is still some 
confusion out there, but the gist from surfing the various articles (and 
trimming away the hysteria) appears to be:

* there are two security issues, both stemming from the same root cause. The 
“problem” has actually been around for nearly 20 years, but faster processors 
are making it much more visible.

* one problem (Meltdown) specifically impacts at least Intel, ARM, and AMD 
processors. This problem is the one that the kernel patches address as it can 
be corrected via software, albeit with some impact that varies based on 
application. Those apps that perform lots of kernel services will see larger 
impacts than those that don’t use the kernel much.

* the other problem (Spectre) appears to impact _all_ processors (including, by 
some reports, SPARC and Power). This problem lacks a software solution

* the “problem” is only a problem if you are running on shared nodes - i.e., if 
multiple users share a common OS instance as it allows a user to potentially 
access the kernel information of the other user. So HPC installations that 
allocate complete nodes to a single user might want to take a closer look 
before installing the patches. Ditto for your desktop and laptop - unless 
someone can gain access to the machine, it isn’t really a “problem”.

* containers and VMs don’t fully resolve the problem - the only solution other 
than the patches is to limit allocations to single users on a node

HTH
Ralph


> On Jan 3, 2018, at 10:47 AM, r...@open-mpi.org wrote:
> 
> Well, it appears from that article that the primary impact comes from 
> accessing kernel services. With an OS-bypass network, that shouldn’t happen 
> all that frequently, and so I would naively expect the impact to be at the 
> lower end of the reported scale for those environments. TCP-based systems, 
> though, might be on the other end.
> 
> Probably something we’ll only really know after testing.
> 
>> On Jan 3, 2018, at 10:24 AM, Noam Bernstein <noam.bernst...@nrl.navy.mil> 
>> wrote:
>> 
>> Out of curiosity, have any of the OpenMPI developers tested (or care to 
>> speculate) how strongly affected OpenMPI based codes (just the MPI part, 
>> obviously) will be by the proposed Intel CPU memory-mapping-related kernel 
>> patches that are all the rage?
>> 
>>  
>> https://arstechnica.com/gadgets/2018/01/whats-behind-the-intel-design-flaw-forcing-numerous-patches/
>> 
>>  
>> Noam
>> ___
>> users mailing list
>> users@lists.open-mpi.org
>> https://lists.open-mpi.org/mailman/listinfo/users
> 
> ___
> users mailing list
> users@lists.open-mpi.org
> https://lists.open-mpi.org/mailman/listinfo/users

___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users

Re: [OMPI users] latest Intel CPU bug

[r...@open-mpi.org]

Well, it appears from that article that the primary impact comes from accessing 
kernel services. With an OS-bypass network, that shouldn’t happen all that 
frequently, and so I would naively expect the impact to be at the lower end of 
the reported scale for those environments. TCP-based systems, though, might be 
on the other end.

Probably something we’ll only really know after testing.

> On Jan 3, 2018, at 10:24 AM, Noam Bernstein  
> wrote:
> 
> Out of curiosity, have any of the OpenMPI developers tested (or care to 
> speculate) how strongly affected OpenMPI based codes (just the MPI part, 
> obviously) will be by the proposed Intel CPU memory-mapping-related kernel 
> patches that are all the rage?
> 
>   
> https://arstechnica.com/gadgets/2018/01/whats-behind-the-intel-design-flaw-forcing-numerous-patches/
> 
>   
> Noam
> ___
> users mailing list
> users@lists.open-mpi.org
> https://lists.open-mpi.org/mailman/listinfo/users

___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users

Re: [OMPI users] Q: Binding to cores on AWS?

[r...@open-mpi.org]

I honestly don’t know - will have to defer to Brian, who is likely out for at 
least the extended weekend. I’ll point this one to him when he returns.


> On Dec 22, 2017, at 1:08 PM, Brian Dobbins <bdobb...@gmail.com> wrote:
> 
> 
>   Hi Ralph,
> 
>   OK, that certainly makes sense - so the next question is, what prevents 
> binding memory to be local to particular cores?  Is this possible in a 
> virtualized environment like AWS HVM instances?
> 
>   And does this apply only to dynamic allocations within an instance, or 
> static as well?  I'm pretty unfamiliar with how the hypervisor (KVM-based, I 
> believe) maps out 'real' hardware, including memory, to particular instances. 
>  We've seen some parts of the code (bandwidth heavy) run ~10x faster on 
> bare-metal hardware, though, presumably from memory locality, so it certainly 
> has a big impact.
> 
>   Thanks again, and merry Christmas!
>   - Brian
> 
> 
> On Fri, Dec 22, 2017 at 1:53 PM, r...@open-mpi.org <mailto:r...@open-mpi.org> 
> <r...@open-mpi.org <mailto:r...@open-mpi.org>> wrote:
> Actually, that message is telling you that binding to core is available, but 
> that we cannot bind memory to be local to that core. You can verify the 
> binding pattern by adding --report-bindings to your cmd line.
> 
> 
>> On Dec 22, 2017, at 11:58 AM, Brian Dobbins <bdobb...@gmail.com 
>> <mailto:bdobb...@gmail.com>> wrote:
>> 
>> 
>> Hi all,
>> 
>>   We're testing a model on AWS using C4/C5 nodes and some of our timers, in 
>> a part of the code with no communication, show really poor performance 
>> compared to native runs.  We think this is because we're not binding to a 
>> core properly and thus not caching, and a quick 'mpirun --bind-to core 
>> hostname' does suggest issues with this on AWS:
>> 
>> [bdobbins@head run]$ mpirun --bind-to core hostname
>> --
>> WARNING: a request was made to bind a process. While the system
>> supports binding the process itself, at least one node does NOT
>> support binding memory to the process location.
>> 
>>   Node:  head
>> 
>> Open MPI uses the "hwloc" library to perform process and memory
>> binding. This error message means that hwloc has indicated that
>> processor binding support is not available on this machine.
>> 
>>   (It also happens on compute nodes, and with real executables.)
>> 
>>   Does anyone know how to enforce binding to cores on AWS instances?  Any 
>> insight would be great.  
>> 
>>   Thanks,
>>   - Brian
>> ___
>> users mailing list
>> users@lists.open-mpi.org <mailto:users@lists.open-mpi.org>
>> https://lists.open-mpi.org/mailman/listinfo/users 
>> <https://lists.open-mpi.org/mailman/listinfo/users>
> 
> ___
> users mailing list
> users@lists.open-mpi.org <mailto:users@lists.open-mpi.org>
> https://lists.open-mpi.org/mailman/listinfo/users 
> <https://lists.open-mpi.org/mailman/listinfo/users>
> 
> ___
> users mailing list
> users@lists.open-mpi.org
> https://lists.open-mpi.org/mailman/listinfo/users

___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users

Re: [OMPI users] Q: Binding to cores on AWS?

[r...@open-mpi.org]

Actually, that message is telling you that binding to core is available, but 
that we cannot bind memory to be local to that core. You can verify the binding 
pattern by adding --report-bindings to your cmd line.


> On Dec 22, 2017, at 11:58 AM, Brian Dobbins  wrote:
> 
> 
> Hi all,
> 
>   We're testing a model on AWS using C4/C5 nodes and some of our timers, in a 
> part of the code with no communication, show really poor performance compared 
> to native runs.  We think this is because we're not binding to a core 
> properly and thus not caching, and a quick 'mpirun --bind-to core hostname' 
> does suggest issues with this on AWS:
> 
> [bdobbins@head run]$ mpirun --bind-to core hostname
> --
> WARNING: a request was made to bind a process. While the system
> supports binding the process itself, at least one node does NOT
> support binding memory to the process location.
> 
>   Node:  head
> 
> Open MPI uses the "hwloc" library to perform process and memory
> binding. This error message means that hwloc has indicated that
> processor binding support is not available on this machine.
> 
>   (It also happens on compute nodes, and with real executables.)
> 
>   Does anyone know how to enforce binding to cores on AWS instances?  Any 
> insight would be great.  
> 
>   Thanks,
>   - Brian
> ___
> users mailing list
> users@lists.open-mpi.org
> https://lists.open-mpi.org/mailman/listinfo/users

___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users

Re: [OMPI users] openmpi-v3.0.0: error for --map-by

[r...@open-mpi.org]

Nope - works that way too (running from rhc001):

$ mpirun -H rhc002:24 --map-by ppr:1:socket:pe=1 date
Wed Dec 20 02:08:18 PST 2017
Wed Dec 20 02:08:18 PST 2017
$


> On Dec 20, 2017, at 1:51 AM, Siegmar Gross 
> <siegmar.gr...@informatik.hs-fulda.de> wrote:
> 
> Hi Ralph,
> 
> the problem occurs when you add --host with a different machine. Without
> --host or with "--host " everything works well.
> 
> pc03 fd1026 111 which mpiexec
> /usr/local/openmpi-3.0.0_64_gcc/bin/mpiexec
> 
> pc03 fd1026 112 mpiexec -np 2 --map-by ppr:1:socket:pe=1 date
> [pc03:09373] SETTING BINDING TO CORE
> Wed Dec 20 10:44:21 CET 2017
> Wed Dec 20 10:44:21 CET 2017
> 
> pc03 fd1026 113 mpiexec -np 2 --host pc03:2 --map-by ppr:1:socket:pe=1 date
> [pc03:09385] SETTING BINDING TO CORE
> Wed Dec 20 10:44:43 CET 2017
> Wed Dec 20 10:44:43 CET 2017
> 
> pc03 fd1026 114 mpiexec -np 2 --host pc02:2 --map-by ppr:1:socket:pe=1 date
> [pc03:09395] SETTING BINDING TO CORE
> [pc02:08340] SETTING BINDING TO CORE
> --
> The request to bind processes could not be completed due to
> an internal error - the locale of the following process was
> not set by the mapper code:
> ...
> 
> 
> Kind regards
> 
> Siegmar
> 
> 
> On 12/20/17 09:22, r...@open-mpi.org <mailto:r...@open-mpi.org> wrote:
>> I just checked the head of both the master and 3.0.x branches, and they both 
>> work fine:
>> $ mpirun  --map-by ppr:1:socket:pe=1 date
>> [rhc001:139231] SETTING BINDING TO CORE
>> [rhc002.cluster:203672] SETTING BINDING TO CORE
>> Wed Dec 20 00:20:55 PST 2017
>> Wed Dec 20 00:20:55 PST 2017
>> Tue Dec 19 18:37:03 PST 2017
>> Tue Dec 19 18:37:03 PST 2017
>> $
>> I’ll remove the debug, but it looks like this was already fixed.
>>> On Dec 19, 2017, at 10:49 PM, Siegmar Gross 
>>> <siegmar.gr...@informatik.hs-fulda.de 
>>> <mailto:siegmar.gr...@informatik.hs-fulda.de> 
>>> <mailto:siegmar.gr...@informatik.hs-fulda.de 
>>> <mailto:siegmar.gr...@informatik.hs-fulda.de>>> wrote:
>>> 
>>> Hi,
>>> 
>>> I've installed openmpi-v3.0.0 on my "SUSE Linux Enterprise Server 12.3 
>>> (x86_64)" with gcc-6.4.0. Today I discovered that I get an error for 
>>> --map-by that I don't
>>> get with older versions.
>>> 
>>> 
>>> loki fd1026 115 which mpiexec
>>> /usr/local/openmpi-2.0.3_64_gcc/bin/mpiexec
>>> loki fd1026 116 mpiexec --host pc02:2,pc03:2 --map-by ppr:1:socket:pe=1 date
>>> Wed Dec 20 07:41:00 CET 2017
>>> ,...
>>> 
>>> loki fd1026 107 which mpiexec
>>> /usr/local/openmpi-2.1.2_64_gcc/bin/mpiexec
>>> loki fd1026 108 mpiexec --host pc02:2,pc03:2 --map-by ppr:1:socket:pe=1 date
>>> Wed Dec 20 07:41:27 CET 2017
>>> ...
>>> 
>>> loki fd1026 107 which mpiexec
>>> /usr/local/openmpi-3.0.0_64_gcc/bin/mpiexec
>>> loki fd1026 108 mpiexec --host pc02:2,pc03:2 --map-by ppr:1:socket:pe=1 date
>>> [loki:32662] SETTING BINDING TO CORE
>>> [pc02:04420] SETTING BINDING TO CORE
>>> [pc03:04788] SETTING BINDING TO CORE
>>> --
>>> The request to bind processes could not be completed due to
>>> an internal error - the locale of the following process was
>>> not set by the mapper code:
>>> 
>>>  Process:  [[57386,1],3]
>>> 
>>> Please contact the OMPI developers for assistance. Meantime,
>>> you will still be able to run your application without binding
>>> by specifying "--bind-to none" on your command line.
>>> --
>>> --
>>> ORTE has lost communication with a remote daemon.
>>> 
>>>  HNP daemon   : [[57386,0],0] on node loki
>>>  Remote daemon: [[57386,0],2] on node pc03
>>> 
>>> This is usually due to either a failure of the TCP network
>>> connection to the node, or possibly an internal failure of
>>> the daemon itself. We cannot recover from this failure, and
>>> therefore will terminate the job.
>>> --
>>> [loki:32662] 1 more process has sent help message help-orte-rmaps-base.txt 
>>> / rmaps:no-locale
>>> [loki:32662] Set MCA parameter "orte_base_help_aggregate" to 0 to se

Re: [OMPI users] openmpi-v3.0.0: error for --map-by

[r...@open-mpi.org]

I just checked the head of both the master and 3.0.x branches, and they both 
work fine:

$ mpirun  --map-by ppr:1:socket:pe=1 date
[rhc001:139231] SETTING BINDING TO CORE
[rhc002.cluster:203672] SETTING BINDING TO CORE
Wed Dec 20 00:20:55 PST 2017
Wed Dec 20 00:20:55 PST 2017
Tue Dec 19 18:37:03 PST 2017
Tue Dec 19 18:37:03 PST 2017
$

I’ll remove the debug, but it looks like this was already fixed.

> On Dec 19, 2017, at 10:49 PM, Siegmar Gross 
>  wrote:
> 
> Hi,
> 
> I've installed openmpi-v3.0.0 on my "SUSE Linux Enterprise Server 12.3 
> (x86_64)" with gcc-6.4.0. Today I discovered that I get an error for --map-by 
> that I don't
> get with older versions.
> 
> 
> loki fd1026 115 which mpiexec
> /usr/local/openmpi-2.0.3_64_gcc/bin/mpiexec
> loki fd1026 116 mpiexec --host pc02:2,pc03:2 --map-by ppr:1:socket:pe=1 date
> Wed Dec 20 07:41:00 CET 2017
> ,...
> 
> loki fd1026 107 which mpiexec
> /usr/local/openmpi-2.1.2_64_gcc/bin/mpiexec
> loki fd1026 108 mpiexec --host pc02:2,pc03:2 --map-by ppr:1:socket:pe=1 date
> Wed Dec 20 07:41:27 CET 2017
> ...
> 
> loki fd1026 107 which mpiexec
> /usr/local/openmpi-3.0.0_64_gcc/bin/mpiexec
> loki fd1026 108 mpiexec --host pc02:2,pc03:2 --map-by ppr:1:socket:pe=1 date
> [loki:32662] SETTING BINDING TO CORE
> [pc02:04420] SETTING BINDING TO CORE
> [pc03:04788] SETTING BINDING TO CORE
> --
> The request to bind processes could not be completed due to
> an internal error - the locale of the following process was
> not set by the mapper code:
> 
>  Process:  [[57386,1],3]
> 
> Please contact the OMPI developers for assistance. Meantime,
> you will still be able to run your application without binding
> by specifying "--bind-to none" on your command line.
> --
> --
> ORTE has lost communication with a remote daemon.
> 
>  HNP daemon   : [[57386,0],0] on node loki
>  Remote daemon: [[57386,0],2] on node pc03
> 
> This is usually due to either a failure of the TCP network
> connection to the node, or possibly an internal failure of
> the daemon itself. We cannot recover from this failure, and
> therefore will terminate the job.
> --
> [loki:32662] 1 more process has sent help message help-orte-rmaps-base.txt / 
> rmaps:no-locale
> [loki:32662] Set MCA parameter "orte_base_help_aggregate" to 0 to see all 
> help / error messages
> loki fd1026 109
> 
> 
> 
> I would be grateful, if somebody can fix the problem. Do you need anything
> else? Thank you very much for any help in advance.
> 
> 
> Kind regards
> 
> Siegmar
> ___
> users mailing list
> users@lists.open-mpi.org
> https://lists.open-mpi.org/mailman/listinfo/users

___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users

Re: [OMPI users] OpenMPI & Slurm: mpiexec/mpirun vs. srun

[r...@open-mpi.org]

> On Dec 19, 2017, at 8:46 AM, Charles A Taylor <chas...@ufl.edu> wrote:
> 
> Hi All,
> 
> I’m glad to see this come up.  We’ve used OpenMPI for a long time and 
> switched to SLURM (from torque+moab) about 2.5 years ago.  At the time, I had 
> a lot of questions about running MPI jobs under SLURM and good information 
> seemed to be scarce - especially regarding “srun”.   I’ll just briefly share 
> my/our observations.  For those who are interested, there are examples of our 
> suggested submission scripts at 
> https://help.rc.ufl.edu/doc/Sample_SLURM_Scripts#MPI_job 
> <https://help.rc.ufl.edu/doc/Sample_SLURM_Scripts#MPI_job> (as I type this 
> I’m hoping that page is up-to-date).  Feel free to comment or make 
> suggestions if you have had different experiences or know better (very 
> possible).
> 
> 1. We initially ignored srun since mpiexec _seemed_ to work fine (more below).
> 
> 2. We soon started to get user complaints of MPI apps running at 1/2 to 1/3 
> of their expected or previously observed speeds - but only sporadically - 
> meaning that sometimes the same job, submitted the same way would run at full 
> speed and sometimes at 1/2 or 1/3 (almost exactly) speed.
> 
> Investigation showed that some MPI ranks in the job were time-slicing across 
> one or more of the cores allocated by SLURM.  It turns out that if the slurm 
> allocation is not consistent with the default OMPI core/socket mapping, this 
> can easily happen.  It can be avoided by a) using “srun —mpi=pmi2” or as of 
> 2.x, “srun —mpi=pmix” or b) more carefully crafting your slurm resource 
> request to be consistent with the OMPI default core/socket mapping.

Or one could tell OMPI to do what you really want it to do using map-by and 
bind-to options, perhaps putting them in the default MCA param file.

Or you could enable cgroups in slurm so that OMPI sees the binding envelope - 
it will respect it. The problem is that OMPI isn’t seeing the requested binding 
envelope and thinks resources are available that really aren’t, and so it gets 
confused about how to map things. Slurm expresses that envelope in an envar, 
but the name and syntax keep changing over the releases, and we just can’t 
track it all the time.

However, I agree that it can be a problem if Slurm is allocating resources in a 
non-HPC manner (i.e., not colocating allocations to maximize performance) and 
you just want to use the default mpiexec options. We only see that when someone 
configures slurm to not allocate nodes to single users, which is not the normal 
HPC mode of operation.

So if you are going to configure slurm to operate in the “cloud” mode of 
allocating individual processor assets, then yes - probably better to use srun 
instead of the default mpiexec options, or add some directives to the default 
MCA param file.

> 
> So beware of resource requests that specify only the number of tasks 
> (—ntasks=64) and then launch with “mpiexec”.  Slurm will happily allocate 
> those tasks anywhere it can (on a busy cluster) and you will get some very 
> non-optimal core mappings/bindings and, possibly, core sharing.
> 
> 3. While doing some spank development for a local, per-job (not per step) 
> temporary directory, I noticed that when launching multi-host MPI jobs with 
> mpiexec vs srun, you end up with more than one host with “slurm_nodeid=1”.  
> I’m not sure if this is a bug (it was 15.08.x) or not and it didn’t seem to 
> cause issues but I also don’t think that it is ideal for two nodes in the 
> same job to have the some numeric nodeid.   When launching with “srun”, that 
> didn’t happen.

I’m not sure what “slurm_nodeid” is - where does this come from?

> 
> Anyway, that is what we have observed.  Generally speaking, I try to get 
> users to use “srun” but many of them still use “mpiexec” out of habit.  You 
> know what they say about old habits.  

Again, it truly depends on how things are configured, if the users are using 
scripts that need to port to other environments, etc.

> 
> Comments, suggestions, or just other experiences are welcome.  Also, if 
> anyone is interested in the tmpdir spank plugin, you can contact me.  We are 
> happy to share.
> 
> Best and Merry Christmas to all,
> 
> Charlie Taylor
> UF Research Computing
> 
> 
> 
>> On Dec 18, 2017, at 8:12 PM, r...@open-mpi.org <mailto:r...@open-mpi.org> 
>> wrote:
>> 
>> We have had reports of applications running faster when executing under 
>> OMPI’s mpiexec versus when started by srun. Reasons aren’t entirely clear, 
>> but are likely related to differences in mapping/binding options (OMPI 
>> provides a very large range compared to srun) and optimization flags 
>> provided by mpiexec that are specific to OMPI.
>> 
>> O

Re: [OMPI users] OpenMPI & Slurm: mpiexec/mpirun vs. srun

[r...@open-mpi.org]

We have had reports of applications running faster when executing under OMPI’s 
mpiexec versus when started by srun. Reasons aren’t entirely clear, but are 
likely related to differences in mapping/binding options (OMPI provides a very 
large range compared to srun) and optimization flags provided by mpiexec that 
are specific to OMPI.

OMPI uses PMIx for wireup support (starting with the v2.x series), which 
provides a faster startup than other PMI implementations. However, that is also 
available with Slurm starting with the 16.05 release, and some further 
PMIx-based launch optimizations were recently added to the Slurm 17.11 release. 
So I would expect that launch via srun with the latest Slurm release and PMIx 
would be faster than mpiexec - though that still leaves the faster execution 
reports to consider.

HTH
Ralph


> On Dec 18, 2017, at 2:18 PM, Prentice Bisbal  wrote:
> 
> Greeting OpenMPI users and devs!
> 
> We use OpenMPI with Slurm as our scheduler, and a user has asked me this: 
> should they use mpiexec/mpirun or srun to start their MPI jobs through Slurm?
> 
> My inclination is to use mpiexec, since that is the only method that's 
> (somewhat) defined in the MPI standard and therefore the most portable, and 
> the examples in the OpenMPI FAQ use mpirun. However, the Slurm documentation 
> on the schedmd website say to use srun with the --mpi=pmi option. (See links 
> below)
> 
> What are the pros/cons of using these two methods, other than the portability 
> issue I already mentioned? Does srun+pmi use a different method to wire up 
> the connections? Some things I read online seem to indicate that. If slurm 
> was built with PMI support, and OpenMPI was built with Slurm support, does it 
> really make any difference?
> 
> https://www.open-mpi.org/faq/?category=slurm
> https://slurm.schedmd.com/mpi_guide.html#open_mpi
> 
> 
> -- 
> Prentice
> 
> ___
> users mailing list
> users@lists.open-mpi.org
> https://lists.open-mpi.org/mailman/listinfo/users

___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users

Re: [OMPI users] OMPI 3.0.0 crashing at mpi_init on OS X using Fortran

[r...@open-mpi.org]

FWIW: I just cloned the v3.0.x branch to get the latest 3.0.1 release 
candidate, built and ran it on Mac OSX High Sierra. Everything built and ran 
fine for both C and Fortran codes.

You might want to test the same - could be this was already fixed.

> On Dec 11, 2017, at 12:43 PM, Ricardo Parreira de Azambuja Fonseca 
>  wrote:
> 
> Hi guys
> 
> I’m having problems with a Fortran based code that I develop with OpenMPI 
> 3.0.0 on Mac OS X. The problem shows itself with both gfortran and intel 
> ifort compilers, and it runs perfectly with version 2.1.2 (and earlier 
> versions).
> 
> Launching the code, even without using mpiexec, causes a segfault when my 
> code calls mpi_init()
> 
> Program received signal SIGSEGV: Segmentation fault - invalid memory 
> reference.
> 
> Backtrace for this error:
> #0  0x1107a41fc
> (…)
> #10  0x10f86eff1
> Segmentation fault: 11
> 
> Recompiling OpenMPI with —enable-debug, and launching the code through lldb 
> gives:
> 
> (lldb) run
> Process 65169 launched: '../source/build/osiris.e' (x86_64)
> Process 65169 stopped
> * thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS 
> (code=1, address=0x48)
>frame #0: 0x000100fbe79a 
> libmpi.40.dylib`ompi_hook_base_mpi_init_top_post_opal(argc=0, 
> argv=0x, requested=0, provided=0x7ffeefbfe290) at 
> hook_base.c:278
>   275
>   276 void ompi_hook_base_mpi_init_top_post_opal(int argc, char 
> **argv, int requested, int *provided)
>   277 {
> -> 278HOOK_CALL_COMMON( mpi_init_top_post_opal, argc, argv, 
> requested, provided);
>   279 }
>   280
>   281 void ompi_hook_base_mpi_init_bottom(int argc, char **argv, int 
> requested, int *provided)
> Target 0: (osiris.e) stopped.
> (lldb) bt
> * thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS 
> (code=1, address=0x48)
>  * frame #0: 0x000100fbe79a 
> libmpi.40.dylib`ompi_hook_base_mpi_init_top_post_opal(argc=0, 
> argv=0x, requested=0, provided=0x7ffeefbfe290) at 
> hook_base.c:278
>frame #1: 0x000100dce0ff libmpi.40.dylib`ompi_mpi_init(argc=0, 
> argv=0x, requested=0, provided=0x7ffeefbfe290) at 
> ompi_mpi_init.c:486
>frame #2: 0x000100eb3f38 
> libmpi.40.dylib`PMPI_Init(argc=0x7ffeefbfe2d0, argv=0x7ffeefbfe2c8) 
> at pinit.c:66
>frame #3: 0x000100cceb0b 
> libmpi_mpifh.40.dylib`ompi_init_f(ierr=0x7ffeefbfe9f8) at init_f.c:84
>frame #4: 0x000100ccead5 
> libmpi_mpifh.40.dylib`mpi_init_(ierr=0x7ffeefbfe9f8) at init_f.c:65
>frame #5: 0x00014e5a osiris.e`__m_system_MOD_system_init at 
> os-sys-multi.f03:323
>frame #6: 0x00010036edb5 osiris.e`MAIN__ at os-main.f03:36
>frame #7: 0x00010039eff2 osiris.e`main at memory.h:19
>frame #8: 0x7fff6ee7d115 libdyld.dylib`start + 1
> 
> Any thoughts?
> 
> Thanks in advance,
> Ricardo
> 
> —
> Ricardo Fonseca
> 
> Full Professor | Professor Catedrático
> GoLP - Grupo de Lasers e Plasmas
> Instituto de Plasmas e Fusão Nuclear
> Instituto Superior Técnico
> Av. Rovisco Pais
> 1049-001 Lisboa
> Portugal
> 
> tel: +351 21 8419202
> web: http://epp.tecnico.ulisboa.pt/
> ___
> users mailing list
> users@lists.open-mpi.org
> https://lists.open-mpi.org/mailman/listinfo/users

___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users

Re: [OMPI users] OMPI 2.1.2 and SLURM compatibility

[r...@open-mpi.org]

t appears to my naive eye contradictory information.
> Near the top of that file, I find,
> 
> # major, minor, and release are generally combined in the form
> # ...
> 
> major=2
> minor=0
> release=1
> 
> then some greek and a repo version (I think to the pmix repository version)
> 
> greek=rc1
> repo_rev=2.0.1rc1
> 
> That leads me to believe that the bundled version is 2.0.1, Release
> Candidate 1.  Then comes the possibly contradictory part I really
> don't understand:
> 
> 
> # 1. Since these version numbers are associated with *releases*, the
> # version numbers maintained on the PMIx Github trunk (and developer
> # branches) is always 0:0:0 for all libraries.
> 
> # 2. The version number of libpmix refers to the public pmix interfaces.
> # It does not refer to any internal interfaces.
> 
> # Version numbers are described in the Libtool current:revision:age
> # format.
> 
> libpmix_so_version=3:1:1
> 
> 
> The library version is different from the software version?  OK, maybe
> that is true, but that isn't what's being tested by the configure
> script.
> 
> I am further befuddled because, as near as I can tell, the bundled
> version of PMIx that comes with OMPI 3.0.0 will fail that test, too.
> I look in
> 
>openmpi-3.0.0/opal/mca/pmix/pmix2x/pmix/include/pmix_version.h
> 
> and I see
> 
> /* define PMIx version */
> #define PMIX_VERSION_MAJOR 2L
> #define PMIX_VERSION_MINOR 0L
> 
> which is the same as in my installed version.
> 
> The error message I get when I run with a bare `srun ./hello-mpi`
> tells me that I have misconfigured the OMPI build without SLURM's PMI
> support.
> 
> The test that fails to identify the proper version of
> PMIX_VERSION_MAJOR also fails when using the bundled PMIx.
> 
> $ gcc -I/tmp/build/openmpi-3.0.0/opal/mca/pmix/pmix2x/pmix/include \
>   pmix-test.c
> pmix-test.c:95:2: error: #error "not version 3"
> #error "not version 3"
>  ^
> 
> But the config.log generated when using the internal version of PMIx
> seems to completely bypass the test that fails when using an external
> PMIx.
> 
> Shouldn't the test be for PMIX_VERSION_MAJOR != 2L?  The only version
> that has a 3 in it is in the VERSION file at the root of the PMIx
> source, and I don't see that as being used by configure.
> 
> Sorry if this is a long and winding path taken by the ignorant.
> 
> What am I missing?
> 
> Thanks,-- bennet
> 
> 
> 
> On Wed, Nov 29, 2017 at 9:53 AM, r...@open-mpi.org <r...@open-mpi.org> wrote:
>> Hi Bennet
>> 
>> I suspect the problem here lies in the slurm PMIx plugin. Slurm 17.11 
>> supports PMIx v2.0 as well as (I believe) PMIx v1.2. I’m not sure if slurm 
>> is somehow finding one of those on your system and building the plugin or 
>> not, but it looks like OMPI is picking up signs of PMIx being active and 
>> trying to use it - and hitting an incompatibility.
>> 
>> You can test this out by adding --mpi=pmi2 to your srun cmd line and see if 
>> that solves the problem (you may also need to add OMPI_MCA_pmix=s2 to your 
>> environment as slurm has a tendency to publish envars even when they aren’t 
>> being used).
>> 
>> 
>> 
>>> On Nov 29, 2017, at 5:44 AM, Bennet Fauber <ben...@umich.edu> wrote:
>>> 
>>> Howard,
>>> 
>>> Thanks very much for the help identifying what information I should provide.
>>> 
>>> This is some information about our SLURM version
>>> 
>>> $ srun --mpi list
>>> srun: MPI types are...
>>> srun: pmi2
>>> srun: pmix_v1
>>> srun: openmpi
>>> srun: pmix
>>> srun: none
>>> 
>>> $ srun --version
>>> slurm 17.11.0-0rc3
>>> 
>>> This is the output from my build script, which should show all the
>>> configure options I used.
>>> 
>>> Checking compilers and things
>>> OMPI is ompi
>>> COMP_NAME is gcc_4_8_5
>>> SRC_ROOT is /sw/src/arcts
>>> PREFIX_ROOT is /sw/arcts/centos7/apps
>>> PREFIX is /sw/arcts/centos7/apps/gcc_4_8_5/openmpi/2.1.2
>>> CONFIGURE_FLAGS are --disable-dlopen --enable-shared
>>> COMPILERS are CC=gcc CXX=g++ FC=gfortran F77=gfortran
>>> No modules loaded
>>> gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-11)
>>> Copyright (C) 2015 Free Software Foundation, Inc.
>>> This is free software; see the source for copying conditions.  There is NO
>>> warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
>>> 
>>> ./configure
>>>

Re: [OMPI users] OMPI 2.1.2 and SLURM compatibility

[r...@open-mpi.org]

slurm
> #!/bin/bash
> #SBATCH -J JOBNAME
> #SBATCH --mail-user=ben...@umich.edu
> #SBATCH --mail-type=NONE
> 
> #SBATCH -N 2
> #SBATCH --ntasks-per-node=1
> #SBATCH --mem-per-cpu=1g
> #SBATCH --cpus-per-task=1
> #SBATCH -A hpcstaff
> #SBATCH -p standard
> 
> #Your code here
> 
> cd /home/bennet/hello
> srun ./hello-mpi
> -
> 
> The results are attached as slurm-114.out, where it looks to me like
> it is trying to invoke pmi2 instead of pmix.
> 
> If I use `srun --mpi pmix ./hello-mpi` in the file submitted to SLURM,
> I get a core dump.
> 
> [bn1.stage.arc-ts.umich.edu:34722] PMIX ERROR: BAD-PARAM in file
> src/dstore/pmix_esh.c at line 996
> [bn2.stage.arc-ts.umich.edu:04597] PMIX ERROR: BAD-PARAM in file
> src/dstore/pmix_esh.c at line 996
> [bn1:34722] *** Process received signal ***
> [bn1:34722] Signal: Segmentation fault (11)
> [bn1:34722] Signal code: Invalid permissions (2)
> [bn1:34722] Failing at address: 0xcf73a0
> [bn1:34722] [ 0] /usr/lib64/libpthread.so.0(+0xf370)[0x2b2420b1d370]
> [bn1:34722] [ 1] [0xcf73a0]
> [bn1:34722] *** End of error message ***
> [bn2:04597] *** Process received signal ***
> [bn2:04597] Signal: Segmentation fault (11)
> [bn2:04597] Signal code:  (128)
> [bn2:04597] Failing at address: (nil)
> [bn2:04597] [ 0] /usr/lib64/libpthread.so.0(+0xf370)[0x2ab526447370]
> [bn2:04597] [ 1]
> /sw/arcts/centos7/apps/gcc_4_8_5/openmpi/2.1.2/lib/libopen-pal.so.20(+0x12291b)[0x2ab52706291b]
> [bn2:04597] [ 2]
> /sw/arcts/centos7/apps/gcc_4_8_5/openmpi/2.1.2/lib/libopen-pal.so.20(PMIx_Init+0x82)[0x2ab527052e32]
> [bn2:04597] [ 3]
> /sw/arcts/centos7/apps/gcc_4_8_5/openmpi/2.1.2/lib/libopen-pal.so.20(pmix1_client_init+0x62)[0x2ab52703b052]
> [bn2:04597] [ 4]
> /sw/arcts/centos7/apps/gcc_4_8_5/openmpi/2.1.2/lib/libopen-rte.so.20(+0x6786b)[0x2ab526c9686b]
> [bn2:04597] [ 5]
> /sw/arcts/centos7/apps/gcc_4_8_5/openmpi/2.1.2/lib/libopen-rte.so.20(orte_init+0x225)[0x2ab526c54165]
> [bn2:04597] [ 6]
> /sw/arcts/centos7/apps/gcc_4_8_5/openmpi/2.1.2/lib/libmpi.so.20(ompi_mpi_init+0x305)[0x2ab5260ab4d5]
> [bn2:04597] [ 7]
> /sw/arcts/centos7/apps/gcc_4_8_5/openmpi/2.1.2/lib/libmpi.so.20(MPI_Init+0x83)[0x2ab5260c95b3]
> [bn2:04597] [ 8] /home/bennet/hello/./hello-mpi[0x4009d5]
> [bn2:04597] [ 9] /usr/lib64/libc.so.6(__libc_start_main+0xf5)[0x2ab526675b35]
> [bn2:04597] [10] /home/bennet/hello/./hello-mpi[0x4008d9]
> [bn2:04597] *** End of error message ***
> srun: error: bn1: task 0: Segmentation fault (core dumped)
> srun: error: bn2: task 1: Segmentation fault (core dumped)
> 
> 
> If I use `srun --mpi openmpi` in the submit script, the job hangs, and
> when I cancel it, I get
> 
> [bn2.stage.arc-ts.umich.edu:04855] PMI_Init [pmix_s1.c:162:s1_init]:
> PMI is not initialized
> [bn1.stage.arc-ts.umich.edu:35000] PMI_Init [pmix_s1.c:162:s1_init]:
> PMI is not initialized
> [warn] opal_libevent2022_event_active: event has no event_base set.
> [warn] opal_libevent2022_event_active: event has no event_base set.
> slurmstepd: error: *** STEP 116.0 ON bn1 CANCELLED AT 2017-11-29T08:42:54 ***
> srun: Job step aborted: Waiting up to 32 seconds for job step to finish.
> slurmstepd: error: *** JOB 116 ON bn1 CANCELLED AT 2017-11-29T08:42:54 ***
> 
> Any thoughts you might have on this would be very much appreciated.
> 
> Thanks,  -- bennet
> 
> 
> 
> On Fri, Nov 17, 2017 at 11:45 PM, Howard Pritchard <hpprit...@gmail.com> 
> wrote:
>> Hello Bennet,
>> 
>> What you are trying to do using srun as the job launcher should work.  Could
>> you post the contents
>> of /etc/slurm/slurm.conf for your system?
>> 
>> Could you also post the output of the following command:
>> 
>> ompi_info --all | grep pmix
>> 
>> to the mail list.
>> 
>> the config.log from your build would also be useful.
>> 
>> Howard
>> 
>> 
>> 2017-11-16 9:30 GMT-07:00 r...@open-mpi.org <r...@open-mpi.org>:
>>> 
>>> What Charles said was true but not quite complete. We still support the
>>> older PMI libraries but you likely have to point us to wherever slurm put
>>> them.
>>> 
>>> However,we definitely recommend using PMIx as you will get a faster launch
>>> 
>>> Sent from my iPad
>>> 
>>>> On Nov 16, 2017, at 9:11 AM, Bennet Fauber <ben...@umich.edu> wrote:
>>>> 
>>>> Charlie,
>>>> 
>>>> Thanks a ton!  Yes, we are missing two of the three steps.
>>>> 
>>>> Will report back after we get pmix installed and after we rebuild
>>>> Slurm.  We do have a new enough versio

Re: [OMPI users] signal handling with mpirun

[r...@open-mpi.org]

Try upgrading to the v3.0, or at least to the latest in the v2.x series. The 
v1.10 series is legacy and no longer maintained.

> On Nov 21, 2017, at 8:20 AM, Kulshrestha, Vipul 
>  wrote:
> 
> Hi,
>  
> I am finding that on Ctrl-C, mpirun immediately stops and does not sends 
> SIGTERM to the child processes.
>  
> I am using openmpi 1.10.6.
>  
> The child processes are able to handle SIGINT. I verified that by a printf in 
> my signal handler and then issuing SIGINT to my process directly. However, 
> when I send the same signal to mpirun process (or press Ctrl-C), it stops 
> right away and my processes do not receive any signal.
>  
> I suspect that I may be doing something wrong, but need some help to 
> troubleshoot this.
>  
> Thanks,
> Vipul
>  
> ___
> users mailing list
> users@lists.open-mpi.org 
> https://lists.open-mpi.org/mailman/listinfo/users 
> 
___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users

Re: [OMPI users] --map-by

[r...@open-mpi.org]

I believe that map-by core with a PE > 1 may have worked at some point in the 
past, but the docs should probably be looked at. I took a (very brief) look at 
the code and re-enabling that particular option would be difficult and not 
really necessary since one can reproduce the desired pattern within the current 
context.

> On Nov 21, 2017, at 5:34 AM, Noam Bernstein <noam.bernst...@nrl.navy.mil> 
> wrote:
> 
>> 
>> On Nov 20, 2017, at 7:02 PM, r...@open-mpi.org <mailto:r...@open-mpi.org> 
>> wrote:
>> 
>> So there are two options here that will work and hopefully provide you with 
>> the desired pattern:
>> 
>> * if you want the procs to go in different NUMA regions:
>> $ mpirun --map-by numa:PE=2 --report-bindings -n 2 /bin/true
>> [rhc001:131460] MCW rank 0 bound to socket 0[core 0[hwt 0-1]], socket 0[core 
>> 1[hwt 0-1]]: 
>> [BB/BB/../../../../../../../../../..][../../../../../../../../../../../..]
>> [rhc001:131460] MCW rank 1 bound to socket 1[core 12[hwt 0-1]], socket 
>> 1[core 13[hwt 0-1]]: 
>> [../../../../../../../../../../../..][BB/BB/../../../../../../../../../..]
>> 
>> * if you want the procs to go in the same NUMA region:
>> $ mpirun --map-by ppr:2:numa:PE=2 --report-bindings -n 2 /bin/true
>> [rhc001:131559] MCW rank 0 bound to socket 0[core 0[hwt 0-1]], socket 0[core 
>> 1[hwt 0-1]]: 
>> [BB/BB/../../../../../../../../../..][../../../../../../../../../../../..]
>> [rhc001:131559] MCW rank 1 bound to socket 0[core 2[hwt 0-1]], socket 0[core 
>> 3[hwt 0-1]]: 
>> [../../BB/BB/../../../../../../../..][../../../../../../../../../../../..]
>> 
>> Reason: the level you are mapping by (e.g., NUMA) must have enough cores in 
>> it to meet your PE=N directive. If you map by core, then there is only one 
>> core in that object.
> 
> Makes sense.  I’ll try that.  However, if I understand your explanation 
> correctly the docs should probably be changed, because they seem to be 
> suggesting something that will never work.   In fact, would the ":PE=N" > 1 
> ever work for "—map-by core”?  I guess maybe if you have hyperthreading on, 
> but I’d still argue that that’s an unhelpful example, given how rarely 
> hyperthreading is used in HPC.
> 
>   Noam
> 
> 
> 
> 
> 
> ||
> |U.S. NAVAL|
> |_RESEARCH_|
> LABORATORY
> 
> Noam Bernstein, Ph.D.
> Center for Materials Physics and Technology
> U.S. Naval Research Laboratory
> T +1 202 404 8628  F +1 202 404 7546
> https://www.nrl.navy.mil <https://www.nrl.navy.mil/>
> ___
> users mailing list
> users@lists.open-mpi.org
> https://lists.open-mpi.org/mailman/listinfo/users

___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users

Re: [OMPI users] --map-by

[r...@open-mpi.org]

So there are two options here that will work and hopefully provide you with the 
desired pattern:

* if you want the procs to go in different NUMA regions:
$ mpirun --map-by numa:PE=2 --report-bindings -n 2 /bin/true
[rhc001:131460] MCW rank 0 bound to socket 0[core 0[hwt 0-1]], socket 0[core 
1[hwt 0-1]]: 
[BB/BB/../../../../../../../../../..][../../../../../../../../../../../..]
[rhc001:131460] MCW rank 1 bound to socket 1[core 12[hwt 0-1]], socket 1[core 
13[hwt 0-1]]: 
[../../../../../../../../../../../..][BB/BB/../../../../../../../../../..]

* if you want the procs to go in the same NUMA region:
$ mpirun --map-by ppr:2:numa:PE=2 --report-bindings -n 2 /bin/true
[rhc001:131559] MCW rank 0 bound to socket 0[core 0[hwt 0-1]], socket 0[core 
1[hwt 0-1]]: 
[BB/BB/../../../../../../../../../..][../../../../../../../../../../../..]
[rhc001:131559] MCW rank 1 bound to socket 0[core 2[hwt 0-1]], socket 0[core 
3[hwt 0-1]]: 
[../../BB/BB/../../../../../../../..][../../../../../../../../../../../..]

Reason: the level you are mapping by (e.g., NUMA) must have enough cores in it 
to meet your PE=N directive. If you map by core, then there is only one core in 
that object.

HTH
Ralph

> On Nov 16, 2017, at 7:08 AM, Noam Bernstein <noam.bernst...@nrl.navy.mil> 
> wrote:
> 
> 
>> On Nov 16, 2017, at 9:49 AM, r...@open-mpi.org <mailto:r...@open-mpi.org> 
>> wrote:
>> 
>> Do not include the “bind-to core” option.the mapping directive already 
>> forces that 
> 
> Same error message, unfortunately. And no, I’m not setting a global binding 
> policy, as far as I can tell:
> 
> env | grep OMPI_MCA
> OMPI_MCA_hwloc_base_report_bindings=1
> [compute-7-6:15083] SETTING BINDING TO CORE
> --
> A request for multiple cpus-per-proc was given, but a directive
> was also give to map to an object level that cannot support that
> directive.
> 
> Please specify a mapping level that has more than one cpu, or
> else let us define a default mapping that will allow multiple
> cpus-per-proc.
> --
> 
>   Noam
> 
> 
> 
> ||
> |U.S. NAVAL|
> |_RESEARCH_|
> LABORATORY
> 
> Noam Bernstein, Ph.D.
> Center for Materials Physics and Technology
> U.S. Naval Research Laboratory
> T +1 202 404 8628  F +1 202 404 7546
> https://www.nrl.navy.mil <https://www.nrl.navy.mil/>
> ___
> users mailing list
> users@lists.open-mpi.org
> https://lists.open-mpi.org/mailman/listinfo/users

___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users

Re: [OMPI users] OMPI 2.1.2 and SLURM compatibility

[r...@open-mpi.org]

What Charles said was true but not quite complete. We still support the older 
PMI libraries but you likely have to point us to wherever slurm put them.

However,we definitely recommend using PMIx as you will get a faster launch 

Sent from my iPad

> On Nov 16, 2017, at 9:11 AM, Bennet Fauber  wrote:
> 
> Charlie,
> 
> Thanks a ton!  Yes, we are missing two of the three steps.
> 
> Will report back after we get pmix installed and after we rebuild
> Slurm.  We do have a new enough version of it, at least, so we might
> have missed the target, but we did at least hit the barn.  ;-)
> 
> 
> 
>> On Thu, Nov 16, 2017 at 10:54 AM, Charles A Taylor  wrote:
>> Hi Bennet,
>> 
>> Three things...
>> 
>> 1. OpenMPI 2.x requires PMIx in lieu of pmi1/pmi2.
>> 
>> 2. You will need slurm 16.05 or greater built with —with-pmix
>> 
>> 2a. You will need pmix 1.1.5 which you can get from github.
>> (https://github.com/pmix/tarballs).
>> 
>> 3. then, to launch your mpi tasks on the allocated resources,
>> 
>>   srun —mpi=pmix ./hello-mpi
>> 
>> I’m replying to the list because,
>> 
>> a) this information is harder to find than you might think.
>> b) someone/anyone can correct me if I’’m giving a bum steer.
>> 
>> Hope this helps,
>> 
>> Charlie Taylor
>> University of Florida
>> 
>> On Nov 16, 2017, at 10:34 AM, Bennet Fauber  wrote:
>> 
>> I think that OpenMPI is supposed to support SLURM integration such that
>> 
>>   srun ./hello-mpi
>> 
>> should work?  I built OMPI 2.1.2 with
>> 
>> export CONFIGURE_FLAGS='--disable-dlopen --enable-shared'
>> export COMPILERS='CC=gcc CXX=g++ FC=gfortran F77=gfortran'
>> 
>> CMD="./configure \
>>   --prefix=${PREFIX} \
>>   --mandir=${PREFIX}/share/man \
>>   --with-slurm \
>>   --with-pmi \
>>   --with-lustre \
>>   --with-verbs \
>>   $CONFIGURE_FLAGS \
>>   $COMPILERS
>> 
>> I have a simple hello-mpi.c (source included below), which compiles
>> and runs with mpirun, both on the login node and in a job.  However,
>> when I try to use srun in place of mpirun, I get instead a hung job,
>> which upon cancellation produces this output.
>> 
>> [bn2.stage.arc-ts.umich.edu:116377] PMI_Init [pmix_s1.c:162:s1_init]:
>> PMI is not initialized
>> [bn1.stage.arc-ts.umich.edu:36866] PMI_Init [pmix_s1.c:162:s1_init]:
>> PMI is not initialized
>> [warn] opal_libevent2022_event_active: event has no event_base set.
>> [warn] opal_libevent2022_event_active: event has no event_base set.
>> slurmstepd: error: *** STEP 86.0 ON bn1 CANCELLED AT 2017-11-16T10:03:24 ***
>> srun: Job step aborted: Waiting up to 32 seconds for job step to finish.
>> slurmstepd: error: *** JOB 86 ON bn1 CANCELLED AT 2017-11-16T10:03:24 ***
>> 
>> The SLURM web page suggests that OMPI 2.x and later support PMIx, and
>> to use `srun --mpi=pimx`, however that no longer seems to be an
>> option, and using the `openmpi` type isn't working (neither is pmi2).
>> 
>> [bennet@beta-build hello]$ srun --mpi=list
>> srun: MPI types are...
>> srun: mpi/pmi2
>> srun: mpi/lam
>> srun: mpi/openmpi
>> srun: mpi/mpich1_shmem
>> srun: mpi/none
>> srun: mpi/mvapich
>> srun: mpi/mpich1_p4
>> srun: mpi/mpichgm
>> srun: mpi/mpichmx
>> 
>> To get the Intel PMI to work with srun, I have to set
>> 
>>   I_MPI_PMI_LIBRARY=/usr/lib64/libpmi.so
>> 
>> Is there a comparable environment variable that must be set to enable
>> `srun` to work?
>> 
>> Am I missing a build option or misspecifying one?
>> 
>> -- bennet
>> 
>> 
>> Source of hello-mpi.c
>> ==
>> #include 
>> #include 
>> #include "mpi.h"
>> 
>> int main(int argc, char **argv){
>> 
>> int rank;  /* rank of process */
>> int numprocs;  /* size of COMM_WORLD */
>> int namelen;
>> int tag=10;/* expected tag */
>> int message;   /* Recv'd message */
>> char processor_name[MPI_MAX_PROCESSOR_NAME];
>> MPI_Status status; /* status of recv */
>> 
>> /* call Init, size, and rank */
>> MPI_Init(, );
>> MPI_Comm_size(MPI_COMM_WORLD, );
>> MPI_Comm_rank(MPI_COMM_WORLD, );
>> MPI_Get_processor_name(processor_name, );
>> 
>> printf("Process %d on %s out of %d\n", rank, processor_name, numprocs);
>> 
>> if(rank != 0){
>>   MPI_Recv(,/*buffer for message */
>>   1,/*MAX count to recv */
>> MPI_INT,/*type to recv */
>>   0,/*recv from 0 only */
>> tag,/*tag of messgae */
>>  MPI_COMM_WORLD,/*communicator to use */
>> );   /*status object */
>>   printf("Hello from process %d!\n",rank);
>> }
>> else{
>>   /* rank 0 ONLY executes this */
>>   printf("MPI_COMM_WORLD is %d processes big!\n", numprocs);
>>   int x;
>>   for(x=1; x>  MPI_Send(,  /*send x to process x */
>>1,  /*number to send */
>>  MPI_INT,  /*type to send */
>>x,  /*rank to send to */
>>  tag,  /*tag for message */
>>

Re: [OMPI users] --map-by

[r...@open-mpi.org]

Do not include the “bind-to core” option.the mapping directive already forces 
that 

Sent from my iPad

> On Nov 16, 2017, at 7:44 AM, Noam Bernstein  
> wrote:
> 
> Hi all - I’m trying to run mixed MPI/OpenMP, so I ideally want binding of 
> each MPI process to a small set of cores (to allow for the OpenMP threads).   
> From the mpirun docs at 
> https://www.open-mpi.org//doc/current/man1/mpirun.1.php
> I got the example that I thought corresponded to what I want,
> % mpirun ... --map-by core:PE=2 --bind-to core
> So I tried
> mpirun -x OMP_NUM_THREADS --map-by core:PE=4 --bind-to core -np 32   python 
> …..
> 
> However, when I run this (with openmpi 3.0.0 or with 1.8.8) I get the 
> following error:
> A request for multiple cpus-per-proc was given, but a directive
> was also give to map to an object level that cannot support that
> directive.
> 
> Please specify a mapping level that has more than one cpu, or
> else let us define a default mapping that will allow multiple
> cpus-per-proc.
> 
> Am I doing something wrong, or is there a mistake in the docs, and it should 
> bind to something other than core?
> 
>   
> thanks,
>   
> Noam
> 
> 
> ___
> users mailing list
> users@lists.open-mpi.org
> https://lists.open-mpi.org/mailman/listinfo/users
___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users

Re: [OMPI users] Can't connect using MPI Ports

[r...@open-mpi.org]

I did a quick check across the v2.1 and v3.0 OMPI releases and both failed, 
though with different signatures. Looks like a problem in the OMPI dynamics 
integration (i.e., the PMIx library looked like it was doing the right things).

I’d suggest filing an issue on the OMPI github site so someone can address it 
(I don’t work much on OMPI any more, I’m afraid).


> On Nov 9, 2017, at 1:54 AM, Florian Lindner  wrote:
> 
>>> The MPI Ports functionality (chapter 10.4 of MPI 3.1), mainly consisting of 
>>> MPI_Open_port, MPI_Comm_accept and
>>> MPI_Comm_connect is not usuable without running an ompi-server as a third 
>>> process?
>> 
>> Yes, that’s correct. The reason for moving in that direction is that the 
>> resource managers, as they continue to
>> integrate PMIx into them, are going to be providing that third party. This 
>> will make connect/accept much easier to use,
>> and a great deal more scalable.
>> 
>> See https://github.com/pmix/RFCs/blob/master/RFC0003.md for an explanation.
> 
> 
> Ok, thanks for that input. I haven't heard of pmix so far (only as part of 
> some ompi error messages).
> 
> Using ompi-server -d -r 'ompi.connect' I was able to publish and retrieve the 
> port name, however, still no connection
> could be established.
> 
> % mpirun -n 1 --ompi-server "file:ompi.connect" ./a.out A
> Published port 3044605953.0:664448538
> 
> % mpirun -n 1 --ompi-server "file:ompi.connect" ./a.out B
> Looked up port 3044605953.0:664448538
> 
> 
> at this point, both processes hang.
> 
> The code is:
> 
> #include 
> #include 
> #include 
> 
> int main(int argc, char **argv)
> {
>  MPI_Init(, );
>  std::string a(argv[1]);
>  char p[MPI_MAX_PORT_NAME];
>  MPI_Comm icomm;
> 
>  if (a == "A") {
>MPI_Open_port(MPI_INFO_NULL, p);
>MPI_Publish_name("foobar", MPI_INFO_NULL, p);
>printf("Published port %s\n", p);
>MPI_Comm_accept(p, MPI_INFO_NULL, 0, MPI_COMM_WORLD, );
>  }
>  if (a == "B") {
>MPI_Lookup_name("foobar", MPI_INFO_NULL, p);
>printf("Looked up port %s\n", p);
>MPI_Comm_connect(p, MPI_INFO_NULL, 0, MPI_COMM_WORLD, );
>  }
> 
>  MPI_Finalize();
> 
>  return 0;
> }
> 
> 
> 
> Do you have any idea?
> 
> Best,
> Florian
> ___
> users mailing list
> users@lists.open-mpi.org
> https://lists.open-mpi.org/mailman/listinfo/users

___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users

Re: [OMPI users] OpenMPI 1.10.x handling of simultaneous MPI_Abort calls

[r...@open-mpi.org]

I see. Then you understand correctly - we are not going to fix the v1.10 series.

> On Nov 8, 2017, at 10:47 AM, Nikolas Antolin <nanto...@gmail.com> wrote:
> 
> That was not my interpretation. His message said he did not observe the race 
> condition for 2 processes, but did for 6 processes. I observe a failure to 
> exit mpirun around 25-30% of the time with 2 processes, causing an 
> inconsistent hang in both my example program and my larger application.
> 
> -Nik
> 
> On Nov 8, 2017 11:40, "r...@open-mpi.org <mailto:r...@open-mpi.org>" 
> <r...@open-mpi.org <mailto:r...@open-mpi.org>> wrote:
> According to the other reporter, it has been fixed in 1.10.7. I haven’t 
> verified that, but I’d suggest trying it first.
> 
> 
>> On Nov 8, 2017, at 8:26 AM, Nikolas Antolin <nanto...@gmail.com 
>> <mailto:nanto...@gmail.com>> wrote:
>> 
>> Thank you for the replies. Do I understand correctly that since OpenMPI 
>> v1.10 is no longer supported, I am unlikely to see a bug fix for this 
>> without moving to v2.x or v3.x? I am dealing with clusters where the 
>> administrators may be loathe to update packages until it is absolutely 
>> necessary, and want to present them with a complete outlook on the problem.
>> 
>> Thanks,
>> Nik
>> 
>> 2017-11-07 19:00 GMT-07:00 r...@open-mpi.org <mailto:r...@open-mpi.org> 
>> <r...@open-mpi.org <mailto:r...@open-mpi.org>>:
>> Glad to hear it has already been fixed :-)
>> 
>> Thanks!
>> 
>>> On Nov 7, 2017, at 4:13 PM, Tru Huynh <t...@pasteur.fr 
>>> <mailto:t...@pasteur.fr>> wrote:
>>> 
>>> Hi,
>>> 
>>> On Tue, Nov 07, 2017 at 02:05:20PM -0700, Nikolas Antolin wrote:
>>>> Hello,
>>>> 
>>>> In debugging a test of an application, I recently came across odd behavior
>>>> for simultaneous MPI_Abort calls. Namely, while the MPI_Abort was
>>>> acknowledged by the process output, the mpirun process failed to exit. I
>>>> was able to duplicate this behavior on multiple machines with OpenMPI
>>>> versions 1.10.2, 1.10.5, and 1.10.6 with the following simple program:
>>>> 
>>>> #include 
>>>> #include 
>>>> #include 
>>>> #include 
>>>> 
>>>> int main(int argc, char **argv)
>>>> {
>>>>int rank;
>>>> 
>>>>MPI_Init(,);
>>>>MPI_Comm_rank(MPI_COMM_WORLD, );
>>>> 
>>>>printf("I am process number %d\n", rank);
>>>>MPI_Abort(MPI_COMM_WORLD, 3);
>>>>return 0;
>>>> }
>>>> 
>>>> Is this a bug or a feature? Does this behavior exist in OpenMPI versions
>>>> 2.0 and 3.0?
>>> I compiled your test case on CentOS-7 with openmpi 1.10.7/2.1.2 and
>>> 3.0.0 and the program seems to run fine.
>>> 
>>> [tru@borma openmpi-test-abort]$ for i in 1.10.7 2.1.2 3.0.0; do 
>>> 
>>> 
>>>  module purge && module add openmpi/$i && mpicc aa.c -o 
>>> aa-$i && ldd aa-$i; mpirun  -n 2 ./aa-$i ; done 
>>> 
>>>  
>>> linux-vdso.so.1 =>  (0x7ffe115bd000)
>>> libmpi.so.12 => /c7/shared/openmpi/1.10.7/lib/libmpi.so.12 
>>> (0x7f40d7b4a000)
>>> libpthread.so.0 => /lib64/libpthread.so.0 (0x7f40d78f7000)
>>> libc.so.6 => /lib64/libc.so.6 (0x7f40d7534000)
>>> libopen-rte.so.12 => /c7/shared/openmpi/1.10.7/lib/libopen-rte.so.12 
>>> (0x7f40d72b8000)
>>> libopen-pal.so.13 => /c7/shared/openmpi/1.10.7/lib/libopen-pal.so.13 
>>> (0x7f40d6fd9000)
>>> libnuma.so.1 => /lib64/libnuma.so.1 (0x7f40d6dcd000)
>>> libdl.so.2 => /lib64/libdl.so.2 (0x7f40d6bc9000)
>>> librt.so.1 => /lib64/librt.so.1 (0x7f40d69c)
>>> libm.so.6 => /lib64/libm.so.6 (0x7f40d66be000)
>>> libutil.so.1 => /lib64/libutil.so.1 (0x7f40d64bb000)
>>> /lib64/ld-linux-x86-64.so.2 (0x55f6d96c4000)
>>> libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x7f40d62a4000)
>>> I am process number 1
>>> I am process number 0
>>> 

Re: [OMPI users] OpenMPI 1.10.x handling of simultaneous MPI_Abort calls

[r...@open-mpi.org]

According to the other reporter, it has been fixed in 1.10.7. I haven’t 
verified that, but I’d suggest trying it first.


> On Nov 8, 2017, at 8:26 AM, Nikolas Antolin <nanto...@gmail.com> wrote:
> 
> Thank you for the replies. Do I understand correctly that since OpenMPI v1.10 
> is no longer supported, I am unlikely to see a bug fix for this without 
> moving to v2.x or v3.x? I am dealing with clusters where the administrators 
> may be loathe to update packages until it is absolutely necessary, and want 
> to present them with a complete outlook on the problem.
> 
> Thanks,
> Nik
> 
> 2017-11-07 19:00 GMT-07:00 r...@open-mpi.org <mailto:r...@open-mpi.org> 
> <r...@open-mpi.org <mailto:r...@open-mpi.org>>:
> Glad to hear it has already been fixed :-)
> 
> Thanks!
> 
>> On Nov 7, 2017, at 4:13 PM, Tru Huynh <t...@pasteur.fr 
>> <mailto:t...@pasteur.fr>> wrote:
>> 
>> Hi,
>> 
>> On Tue, Nov 07, 2017 at 02:05:20PM -0700, Nikolas Antolin wrote:
>>> Hello,
>>> 
>>> In debugging a test of an application, I recently came across odd behavior
>>> for simultaneous MPI_Abort calls. Namely, while the MPI_Abort was
>>> acknowledged by the process output, the mpirun process failed to exit. I
>>> was able to duplicate this behavior on multiple machines with OpenMPI
>>> versions 1.10.2, 1.10.5, and 1.10.6 with the following simple program:
>>> 
>>> #include 
>>> #include 
>>> #include 
>>> #include 
>>> 
>>> int main(int argc, char **argv)
>>> {
>>>int rank;
>>> 
>>>MPI_Init(,);
>>>MPI_Comm_rank(MPI_COMM_WORLD, );
>>> 
>>>printf("I am process number %d\n", rank);
>>>MPI_Abort(MPI_COMM_WORLD, 3);
>>>return 0;
>>> }
>>> 
>>> Is this a bug or a feature? Does this behavior exist in OpenMPI versions
>>> 2.0 and 3.0?
>> I compiled your test case on CentOS-7 with openmpi 1.10.7/2.1.2 and
>> 3.0.0 and the program seems to run fine.
>> 
>> [tru@borma openmpi-test-abort]$ for i in 1.10.7 2.1.2 3.0.0; do  
>>  
>>  
>>   module purge && module add openmpi/$i && mpicc aa.c -o aa-$i 
>> && ldd aa-$i; mpirun  -n 2 ./aa-$i ; done
>>  
>>  
>>  linux-vdso.so.1 =>  (0x7ffe115bd000)
>>  libmpi.so.12 => /c7/shared/openmpi/1.10.7/lib/libmpi.so.12 
>> (0x7f40d7b4a000)
>>  libpthread.so.0 => /lib64/libpthread.so.0 (0x7f40d78f7000)
>>  libc.so.6 => /lib64/libc.so.6 (0x7f40d7534000)
>>  libopen-rte.so.12 => /c7/shared/openmpi/1.10.7/lib/libopen-rte.so.12 
>> (0x7f40d72b8000)
>>  libopen-pal.so.13 => /c7/shared/openmpi/1.10.7/lib/libopen-pal.so.13 
>> (0x7f40d6fd9000)
>>  libnuma.so.1 => /lib64/libnuma.so.1 (0x7f40d6dcd000)
>>  libdl.so.2 => /lib64/libdl.so.2 (0x7f40d6bc9000)
>>  librt.so.1 => /lib64/librt.so.1 (0x7f40d69c)
>>  libm.so.6 => /lib64/libm.so.6 (0x7f40d66be000)
>>  libutil.so.1 => /lib64/libutil.so.1 (0x7f40d64bb000)
>>  /lib64/ld-linux-x86-64.so.2 (0x55f6d96c4000)
>>  libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x7f40d62a4000)
>> I am process number 1
>> I am process number 0
>> --
>> MPI_ABORT was invoked on rank 1 in communicator MPI_COMM_WORLD 
>> with errorcode 3.
>> 
>> NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
>> You may or may not see output from other processes, depending on
>> exactly when Open MPI kills them.
>> --
>> [borma.bis.pasteur.fr:08511 <http://borma.bis.pasteur.fr:8511/>] 1 more 
>> process has sent help message help-mpi-api.txt / mpi-abort
>> [borma.bis.pasteur.fr:08511 <http://borma.bis.pasteur.fr:8511/>] Set MCA 
>> parameter "orte_base_help_aggregate" to 0 to see all help / error messages
>>  linux-vdso.so.1 =>  (0x7fffaabcd000)
>>  libmpi.so.20 => /c7/shared/openmpi/2.1.2/lib/libmpi.so.20 
>> (0x7f5bcee39000)
>>  libpthread.so.0 => /lib64/libpthread.so.0 (0x7f5bcebe6000)
>>

Re: [OMPI users] Can't connect using MPI Ports

[r...@open-mpi.org]

> On Nov 6, 2017, at 7:46 AM, Florian Lindner <mailingli...@xgm.de> wrote:
> 
> Am 05.11.2017 um 20:57 schrieb r...@open-mpi.org:
>> 
>>> On Nov 5, 2017, at 6:48 AM, Florian Lindner <mailingli...@xgm.de 
>>> <mailto:mailingli...@xgm.de>> wrote:
>>> 
>>> Am 04.11.2017 um 00:05 schrieb r...@open-mpi.org <mailto:r...@open-mpi.org>:
>>>> Yeah, there isn’t any way that is going to work in the 2.x series. I’m not 
>>>> sure it was ever fixed, but you might try
>>>> the latest 3.0, the 3.1rc, and even master.
>>>> 
>>>> The only methods that are known to work are:
>>>> 
>>>> * connecting processes within the same mpirun - e.g., using comm_spawn
>>> 
>>> That is not an option for our application.
>>> 
>>>> * connecting processes across different mpiruns, with the ompi-server 
>>>> daemon as the rendezvous point
>>>> 
>>>> The old command line method (i.e., what you are trying to use) hasn’t been 
>>>> much on the radar. I don’t know if someone
>>>> else has picked it up or not...
>>> 
>>> What do you mean with "the old command line method”.
>>> 
>>> Isn't the ompi-server just another means of exchanging port names, i.e. the 
>>> same I do using files?
>> 
>> No, it isn’t - there is a handshake that ompi-server facilitates.
>> 
>>> 
>>> In my understanding, using Publish_name and Lookup_name or exchanging the 
>>> information using files (or command line or
>>> stdin) shouldn't have any
>>> impact on the connection (Connect / Accept) itself.
>> 
>> Depends on the implementation underneath connect/accept.
>> 
>> The initial MPI standard authors had fixed in their minds that the 
>> connect/accept handshake would take place over a TCP
>> socket, and so no intermediate rendezvous broker was involved. That isn’t 
>> how we’ve chosen to implement it this time
>> around, and so you do need the intermediary. If/when some developer wants to 
>> add another method, they are welcome to do
>> so - but the general opinion was that the broker requirement was fine.
> 
> Ok. Just to make sure I understood correctly:
> 
> The MPI Ports functionality (chapter 10.4 of MPI 3.1), mainly consisting of 
> MPI_Open_port, MPI_Comm_accept and
> MPI_Comm_connect is not usuable without running an ompi-server as a third 
> process?

Yes, that’s correct. The reason for moving in that direction is that the 
resource managers, as they continue to integrate PMIx into them, are going to 
be providing that third party. This will make connect/accept much easier to 
use, and a great deal more scalable.

See https://github.com/pmix/RFCs/blob/master/RFC0003.md 
<https://github.com/pmix/RFCs/blob/master/RFC0003.md> for an explanation.

> 
> Thank again,
> Florian
> ___
> users mailing list
> users@lists.open-mpi.org
> https://lists.open-mpi.org/mailman/listinfo/users

___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users

Re: [OMPI users] Can't connect using MPI Ports

[r...@open-mpi.org]

> On Nov 5, 2017, at 6:48 AM, Florian Lindner <mailingli...@xgm.de> wrote:
> 
> Am 04.11.2017 um 00:05 schrieb r...@open-mpi.org <mailto:r...@open-mpi.org>:
>> Yeah, there isn’t any way that is going to work in the 2.x series. I’m not 
>> sure it was ever fixed, but you might try the latest 3.0, the 3.1rc, and 
>> even master.
>> 
>> The only methods that are known to work are:
>> 
>> * connecting processes within the same mpirun - e.g., using comm_spawn
> 
> That is not an option for our application.
> 
>> * connecting processes across different mpiruns, with the ompi-server daemon 
>> as the rendezvous point
>> 
>> The old command line method (i.e., what you are trying to use) hasn’t been 
>> much on the radar. I don’t know if someone else has picked it up or not...
> 
> What do you mean with "the old command line method”.
> 
> Isn't the ompi-server just another means of exchanging port names, i.e. the 
> same I do using files?

No, it isn’t - there is a handshake that ompi-server facilitates.

> 
> In my understanding, using Publish_name and Lookup_name or exchanging the 
> information using files (or command line or stdin) shouldn't have any
> impact on the connection (Connect / Accept) itself.

Depends on the implementation underneath connect/accept.

The initial MPI standard authors had fixed in their minds that the 
connect/accept handshake would take place over a TCP socket, and so no 
intermediate rendezvous broker was involved. That isn’t how we’ve chosen to 
implement it this time around, and so you do need the intermediary. If/when 
some developer wants to add another method, they are welcome to do so - but the 
general opinion was that the broker requirement was fine.

> 
> Best,
> Florian
> 
> 
>> Ralph
>> 
>>> On Nov 3, 2017, at 11:23 AM, Florian Lindner <mailingli...@xgm.de> wrote:
>>> 
>>> 
>>> Am 03.11.2017 um 16:18 schrieb r...@open-mpi.org:
>>>> What version of OMPI are you using?
>>> 
>>> 2.1.1 @ Arch Linux.
>>> 
>>> Best,
>>> Florian
>>> ___
>>> users mailing list
>>> users@lists.open-mpi.org
>>> https://lists.open-mpi.org/mailman/listinfo/users
>> 
>> ___
>> users mailing list
>> users@lists.open-mpi.org
>> https://lists.open-mpi.org/mailman/listinfo/users
>> 
> ___
> users mailing list
> users@lists.open-mpi.org <mailto:users@lists.open-mpi.org>
> https://lists.open-mpi.org/mailman/listinfo/users 
> <https://lists.open-mpi.org/mailman/listinfo/users>
___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users

Re: [OMPI users] Can't connect using MPI Ports

[r...@open-mpi.org]

Yeah, there isn’t any way that is going to work in the 2.x series. I’m not sure 
it was ever fixed, but you might try the latest 3.0, the 3.1rc, and even master.

The only methods that are known to work are:

* connecting processes within the same mpirun - e.g., using comm_spawn

* connecting processes across different mpiruns, with the ompi-server daemon as 
the rendezvous point

The old command line method (i.e., what you are trying to use) hasn’t been much 
on the radar. I don’t know if someone else has picked it up or not...
Ralph

> On Nov 3, 2017, at 11:23 AM, Florian Lindner <mailingli...@xgm.de> wrote:
> 
> 
> Am 03.11.2017 um 16:18 schrieb r...@open-mpi.org:
>> What version of OMPI are you using?
> 
> 2.1.1 @ Arch Linux.
> 
> Best,
> Florian
> ___
> users mailing list
> users@lists.open-mpi.org
> https://lists.open-mpi.org/mailman/listinfo/users

___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users

Re: [OMPI users] Can't connect using MPI Ports

[r...@open-mpi.org]

What version of OMPI are you using?

> On Nov 3, 2017, at 7:48 AM, Florian Lindner  wrote:
> 
> Hello,
> 
> I'm working on a sample program to connect two MPI communicators launched 
> with mpirun using Ports.
> 
> Firstly, I use MPI_Open_port to obtain a name and write that to a file:
> 
>  if (options.participant == A) { // A publishes the port
>if (options.commType == single and rank == 0)
>  openPublishPort(options);
> 
>if (options.commType == many)
>  openPublishPort(options);
>  }
>  MPI_Barrier(MPI_COMM_WORLD);
> 
> participant is a command line argument and defines the role of A as server. B 
> is the client.
> 
> void openPublishPort(Options options)
> {
>  using namespace boost::filesystem;
>  int rank;
>  MPI_Comm_rank(MPI_COMM_WORLD, );
> 
>  char p[MPI_MAX_PORT_NAME];
>  MPI_Open_port(MPI_INFO_NULL, p);
>  std::string portName(p);
> 
>  create_directory(options.publishDirectory);
>  std::string filename;
>  if (options.commType == many)
>filename = "A-" + std::to_string(rank) + ".address";
>  if (options.commType == single)
>filename = "intercomm.address";
> 
>  auto path = options.publishDirectory / filename;
>  DEBUG << "Writing address " << portName << " to " << path;
>  std::ofstream ofs(path.string(), std::ofstream::out);
>  ofs << portName;
> }
> 
> This works fine as far as I see. Next, I try to connect:
> 
>  MPI_Comm icomm;
>  std::string portName;
>  if (options.participant == A) { // receives connections
>if (options.commType == single) {
>  if (rank == 0)
>portName = readPort(options);
>  INFO << "Accepting connection on " << portName;
>  MPI_Comm_accept(portName.c_str(), MPI_INFO_NULL, 0, MPI_COMM_WORLD, 
> );
>  INFO << "Received connection";
>}
>  }
> 
>  if (options.participant == B) { // connects to the intercomms
>if (options.commType == single) {
>  if (rank == 0)
>portName = readPort(options);
>  INFO << "Trying to connect to " << portName;
>  MPI_Comm_connect(portName.c_str(), MPI_INFO_NULL, 0, MPI_COMM_WORLD, 
> );
>  INFO << "Connected";
>}
>  }
> 
> 
> options.single says that I want to use a single communicator that contains 
> all ranks on both participants, A and B.
> readPort reads the port name from the file that was written before.
> 
> Now, when I first launch A and, in another terminal, B, nothing happens until 
> a timeout occurs.
> 
> % mpirun -n 1 ./mpiports --commType="single" --participant="A"
> [2017-11-03 15:29:55.469891] [debug]   Writing address 
> 3048013825.0:1069313090 to "./publish/intercomm.address"
> [2017-11-03 15:29:55.470169] [debug]   Read address 3048013825.0:1069313090 
> from "./publish/intercomm.address"
> [2017-11-03 15:29:55.470185] [info]Accepting connection on 
> 3048013825.0:1069313090
> [asaru:16199] OPAL ERROR: Timeout in file base/pmix_base_fns.c at line 195
> [...]
> 
> and on the other site:
> 
> % mpirun -n 1 ./mpiports --commType="single" --participant="B"
> [2017-11-03 15:29:59.698921] [debug]   Read address 3048013825.0:1069313090 
> from "./publish/intercomm.address"
> [2017-11-03 15:29:59.698947] [info]Trying to connect to 
> 3048013825.0:1069313090
> [asaru:16238] OPAL ERROR: Timeout in file base/pmix_base_fns.c at line 195
> [...]
> 
> The complete code, including cmake build script can be downloaded at:
> 
> https://www.dropbox.com/s/azo5ti4kjg12zjy/MPI_Ports.tar.gz?dl=0
> 
> Why is the connection not working?
> 
> Thanks a lot,
> Florian
> 
> 
> ___
> users mailing list
> users@lists.open-mpi.org
> https://lists.open-mpi.org/mailman/listinfo/users

___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users

Re: [OMPI users] [OMPI devel] Open MPI 2.0.4rc2 available for testing

[r...@open-mpi.org]

I would suggest also considering simply updating to v3.0, or at least to v2.1. 
I’d rather not play “whack-a-mole” with the Sun compiler again :-(


> On Nov 2, 2017, at 6:06 AM, Howard Pritchard  wrote:
> 
> HI Siegmar,
> 
> Could you check if you also see a similar problem with OMPI master when you 
> build with the Sun compiler?
> 
> I opened issue 4436 to track this issue.  Not sure we'll have time to fix it 
> for 2.0.4 though.
> 
> Howard
> 
> 
> 2017-11-02 3:49 GMT-06:00 Siegmar Gross  >:
> Hi,
> 
> thank you very much for the fix. Unfortunately, I still get an error
> with Sun C 5.15.
> 
> 
> loki openmpi-2.0.4rc2-Linux.x86_64.64_cc 125 tail -30 
> log.make.Linux.x86_64.64_cc
>   CC   src/client/pmix_client.lo
> "/export2/src/openmpi-2.0.4/openmpi-2.0.4rc2/opal/include/opal/sys/x86_64/atomic.h",
>  line 161: warning: parameter in inline asm statement unused: %3
> "/export2/src/openmpi-2.0.4/openmpi-2.0.4rc2/opal/include/opal/sys/x86_64/atomic.h",
>  line 207: warning: parameter in inline asm statement unused: %2
> "/export2/src/openmpi-2.0.4/openmpi-2.0.4rc2/opal/include/opal/sys/x86_64/atomic.h",
>  line 228: warning: parameter in inline asm statement unused: %2
> "/export2/src/openmpi-2.0.4/openmpi-2.0.4rc2/opal/include/opal/sys/x86_64/atomic.h",
>  line 249: warning: parameter in inline asm statement unused: %2
> "/export2/src/openmpi-2.0.4/openmpi-2.0.4rc2/opal/include/opal/sys/x86_64/atomic.h",
>  line 270: warning: parameter in inline asm statement unused: %2
> "../../../../../../openmpi-2.0.4rc2/opal/mca/pmix/pmix112/pmix/src/client/pmix_client.c",
>  line 235: redeclaration must have the same or more restrictive linker 
> scoping: OPAL_PMIX_PMIX112_PMIx_Get_version
> "../../../../../../openmpi-2.0.4rc2/opal/mca/pmix/pmix112/pmix/src/client/pmix_client.c",
>  line 240: redeclaration must have the same or more restrictive linker 
> scoping: OPAL_PMIX_PMIX112_PMIx_Init
> "../../../../../../openmpi-2.0.4rc2/opal/mca/pmix/pmix112/pmix/src/client/pmix_client.c",
>  line 408: redeclaration must have the same or more restrictive linker 
> scoping: OPAL_PMIX_PMIX112_PMIx_Initialized
> "../../../../../../openmpi-2.0.4rc2/opal/mca/pmix/pmix112/pmix/src/client/pmix_client.c",
>  line 416: redeclaration must have the same or more restrictive linker 
> scoping: OPAL_PMIX_PMIX112_PMIx_Finalize
> "../../../../../../openmpi-2.0.4rc2/opal/mca/pmix/pmix112/pmix/src/client/pmix_client.c",
>  line 488: redeclaration must have the same or more restrictive linker 
> scoping: OPAL_PMIX_PMIX112_PMIx_Abort
> "../../../../../../openmpi-2.0.4rc2/opal/mca/pmix/pmix112/pmix/src/client/pmix_client.c",
>  line 616: redeclaration must have the same or more restrictive linker 
> scoping: OPAL_PMIX_PMIX112_PMIx_Put
> "../../../../../../openmpi-2.0.4rc2/opal/mca/pmix/pmix112/pmix/src/client/pmix_client.c",
>  line 703: redeclaration must have the same or more restrictive linker 
> scoping: OPAL_PMIX_PMIX112_PMIx_Commit
> "../../../../../../openmpi-2.0.4rc2/opal/mca/pmix/pmix112/pmix/src/client/pmix_client.c",
>  line 789: redeclaration must have the same or more restrictive linker 
> scoping: OPAL_PMIX_PMIX112_PMIx_Resolve_peers
> "../../../../../../openmpi-2.0.4rc2/opal/mca/pmix/pmix112/pmix/src/client/pmix_client.c",
>  line 852: redeclaration must have the same or more restrictive linker 
> scoping: OPAL_PMIX_PMIX112_PMIx_Resolve_nodes
> cc: acomp failed for 
> ../../../../../../openmpi-2.0.4rc2/opal/mca/pmix/pmix112/pmix/src/client/pmix_client.c
> Makefile:1242: recipe for target 'src/client/pmix_client.lo' failed
> make[4]: *** [src/client/pmix_client.lo] Error 1
> make[4]: Leaving directory 
> '/export2/src/openmpi-2.0.4/openmpi-2.0.4rc2-Linux.x86_64.64_cc/opal/mca/pmix/pmix112/pmix'
> Makefile:1486: recipe for target 'all-recursive' failed
> make[3]: *** [all-recursive] Error 1
> make[3]: Leaving directory 
> '/export2/src/openmpi-2.0.4/openmpi-2.0.4rc2-Linux.x86_64.64_cc/opal/mca/pmix/pmix112/pmix'
> Makefile:1935: recipe for target 'all-recursive' failed
> make[2]: *** [all-recursive] Error 1
> make[2]: Leaving directory 
> '/export2/src/openmpi-2.0.4/openmpi-2.0.4rc2-Linux.x86_64.64_cc/opal/mca/pmix/pmix112'
> Makefile:2301: recipe for target 'all-recursive' failed
> make[1]: *** [all-recursive] Error 1
> make[1]: Leaving directory 
> '/export2/src/openmpi-2.0.4/openmpi-2.0.4rc2-Linux.x86_64.64_cc/opal'
> Makefile:1800: recipe for target 'all-recursive' failed
> make: *** [all-recursive] Error 1
> loki openmpi-2.0.4rc2-Linux.x86_64.64_cc 125
> 
> 
> 
> I would be grateful, if somebody can fix these problems as well.
> Thank you very much for any help in advance.
> 
> 
> Kind regards
> 
> Siegmar
> 
> 
> 
> On 11/01/17 23:18, Howard Pritchard wrote:
> HI Folks,
> 
> We decided to roll an rc2 to pick up a PMIx fix:
> 
> - Fix an issue with visibility of functions defined in the built-in PMIx.
>Thanks to 

Re: [OMPI users] Problem with MPI jobs terminating when using OMPI 3.0.x

[r...@open-mpi.org]

Two questions:

1. are you running this on node04? Or do you have ssh access to node04?

2. I note you are building this against an old version of PMIx for some reason. 
Does it work okay if you build it with the embedded PMIx (which is 2.0)? Does 
it work okay if you use PMIx v1.2.4, the latest release in that series?


> On Oct 27, 2017, at 1:24 PM, Andy Riebs  wrote:
> 
> We have built a version of Open MPI 3.0.x that works with Slurm (our primary 
> use case), but it fails when executed without Slurm.
> 
> If I srun an MPI "hello world" program, it works just fine. Likewise, if I 
> salloc a couple of nodes and use mpirun from there, life is good. But if I 
> just try to mpirun the program without Slurm support, the program appears to 
> run to completion, and then segv's. A bit of good news is that this can be 
> reproduced with a single process.
> 
> Sample output and configuration information below:
> 
> [tests]$ cat gdb.cmd
> set follow-fork-mode child
> r
> [tests]$ mpirun -host node04 -np 1 gdb -x gdb.cmd ./mpi_hello
> GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-94.el7
> Copyright (C) 2013 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later 
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
> and "show warranty" for details.
> This GDB was configured as "x86_64-redhat-linux-gnu".
> For bug reporting instructions, please see:
> ...
> Reading symbols from /home/riebs/tests/mpi_hello...(no debugging symbols 
> found)...done.
> [Thread debugging using libthread_db enabled]
> Using host libthread_db library "/lib64/libthread_db.so.1".
> [New Thread 0x74be8700 (LWP 21386)]
> [New Thread 0x73f70700 (LWP 21387)]
> [New Thread 0x7fffeacac700 (LWP 21393)]
> [Thread 0x7fffeacac700 (LWP 21393) exited]
> [New Thread 0x7fffeacac700 (LWP 21394)]
> Hello world! I'm 0 of 1 on node04
> [Thread 0x7fffeacac700 (LWP 21394) exited]
> [Thread 0x73f70700 (LWP 21387) exited]
> [Thread 0x74be8700 (LWP 21386) exited]
> [Inferior 1 (process 21382) exited normally]
> Missing separate debuginfos, use: debuginfo-install glibc-2.17-157.el7.x86_64 
> libevent-2.0.21-4.el7.x86_64 libgcc-4.8.5-11.el7.x86_6
> 4 libibcm-1.0.5mlnx2-OFED.3.4.0.0.4.34100.x86_64 
> libibumad-1.3.10.2.MLNX20150406.966500d-0.1.34100.x86_64 
> libibverbs-1.2.1mlnx1-OFED
> .3.4.2.1.4.34218.x86_64 libmlx4-1.2.1mlnx1-OFED.3.4.0.0.4.34218.x86_64 
> libmlx5-1.2.1mlnx1-OFED.3.4.2.1.4.34218.x86_64 libnl-1.1.4-3.
> el7.x86_64 librdmacm-1.1.0mlnx-OFED.3.4.0.0.4.34218.x86_64 
> libtool-ltdl-2.4.2-21.el7_2.x86_64 numactl-libs-2.0.9-6.el7_2.x86_64 open
> sm-libs-4.8.0.MLNX20161013.9b1a49b-0.1.34218.x86_64 zlib-1.2.7-17.el7.x86_64
> (gdb) q
> [node04:21373] *** Process received signal ***
> [node04:21373] Signal: Segmentation fault (11)
> [node04:21373] Signal code:  (128)
> [node04:21373] Failing at address: (nil)
> [node04:21373] [ 0] /lib64/libpthread.so.0(+0xf370)[0x760c4370]
> [node04:21373] [ 1] 
> /opt/local/pmix/1.2.1/lib/libpmix.so.2(+0x3a04b)[0x7365104b]
> [node04:21373] [ 2] 
> /lib64/libevent-2.0.so.5(event_base_loop+0x774)[0x764e4a14]
> [node04:21373] [ 3] 
> /opt/local/pmix/1.2.1/lib/libpmix.so.2(+0x285cd)[0x7363f5cd]
> [node04:21373] [ 4] /lib64/libpthread.so.0(+0x7dc5)[0x760bcdc5]
> [node04:21373] [ 5] /lib64/libc.so.6(clone+0x6d)[0x75deb73d]
> [node04:21373] *** End of error message ***
> bash: line 1: 21373 Segmentation fault 
> /opt/local/shmem/3.0.x.4ca1c4d/bin/orted -mca ess "env" -mca ess_base_jobid
> "399966208" -mca ess_base_vpid 1 -mca ess_base_num_procs "2" -mca 
> orte_node_regex "node[2:73],node[4:0]04@0(2)" -mca orte_hnp_uri "3
> 99966208.0;tcp://16.95.253.128,10.4.0.6:52307" -mca plm "rsh" -mca 
> coll_tuned_use_dynamic_rules "1" -mca scoll "^mpi" -mca pml "ucx"
>  -mca coll_tuned_allgatherv_algorithm "2" -mca atomic "ucx" -mca sshmem 
> "mmap" -mca spml_ucx_heap_reg_nb "1" -mca coll_tuned_allgath
> er_algorithm "2" -mca spml "ucx" -mca coll "^hcoll" -mca pmix 
> "^s1,s2,cray,isolated"
> 
> [tests]$ env | grep -E -e MPI -e UCX -e SLURM | sort
> OMPI_MCA_atomic=ucx
> OMPI_MCA_coll=^hcoll
> OMPI_MCA_coll_tuned_allgather_algorithm=2
> OMPI_MCA_coll_tuned_allgatherv_algorithm=2
> OMPI_MCA_coll_tuned_use_dynamic_rules=1
> OMPI_MCA_pml=ucx
> OMPI_MCA_scoll=^mpi
> OMPI_MCA_spml=ucx
> OMPI_MCA_spml_ucx_heap_reg_nb=1
> OMPI_MCA_sshmem=mmap
> OPENMPI_PATH=/opt/local/shmem/3.0.x.4ca1c4d
> OPENMPI_VER=3.0.x.4ca1c4d
> SLURM_DISTRIBUTION=block:block
> SLURM_HINT=nomultithread
> SLURM_SRUN_REDUCE_TASK_EXIT=1
> SLURM_TEST_EXEC=1
> SLURM_UNBUFFEREDIO=1
> SLURM_VER=17.11.0-0pre2
> UCX_TLS=dc_x
> UCX_ZCOPY_THRESH=131072
> [tests]$
> 
> OS: CentOS 7.3
> HW: x86_64 (KNL)
> OMPI version: 3.0.x.4ca1c4d
> Configuration options:
> --prefix=/opt/local/shmem/3.0.x.4ca1c4d
> 

Re: [OMPI users] IpV6 Openmpi mpirun failed

[r...@open-mpi.org]

Actually, I don’t see any related changes in OMPI master, let alone the 
branches. So far as I can tell, the author never actually submitted the work.


> On Oct 19, 2017, at 3:57 PM, Mukkie <mukunthh...@gmail.com> wrote:
> 
> FWIW, my issue is related to this one.
> https://github.com/open-mpi/ompi/issues/1585 
> <https://github.com/open-mpi/ompi/issues/1585>
> 
> I have version 3.0.0 and the above issue is closed saying, fixes went into 
> 3.1.0
> However, i don't see the code changes towards this issue.?
> 
> Cordially,
> Muku.
> 
> On Wed, Oct 18, 2017 at 3:52 PM, Mukkie <mukunthh...@gmail.com 
> <mailto:mukunthh...@gmail.com>> wrote:
> Thanks for your suggestion. However my firewall's are already disabled on 
> both the machines.
> 
> Cordially,
> Muku. 
> 
> On Wed, Oct 18, 2017 at 2:38 PM, r...@open-mpi.org <mailto:r...@open-mpi.org> 
> <r...@open-mpi.org <mailto:r...@open-mpi.org>> wrote:
> Looks like there is a firewall or something blocking communication between 
> those nodes?
> 
>> On Oct 18, 2017, at 1:29 PM, Mukkie <mukunthh...@gmail.com 
>> <mailto:mukunthh...@gmail.com>> wrote:
>> 
>> Adding a verbose output. Please check for failed and advise. Thank you.
>> 
>> [mselvam@ipv-rhel73 examples]$ mpirun -hostfile host --mca oob_base_verbose 
>> 100 --mca btl tcp,self ring_c
>> [ipv-rhel73:10575] mca_base_component_repository_open: unable to open 
>> mca_plm_tm: libtorque.so.2: cannot open shared object file: No such file or 
>> directory (ignored)
>> [ipv-rhel73:10575] mca: base: components_register: registering framework oob 
>> components
>> [ipv-rhel73:10575] mca: base: components_register: found loaded component tcp
>> [ipv-rhel73:10575] mca: base: components_register: component tcp register 
>> function successful
>> [ipv-rhel73:10575] mca: base: components_open: opening oob components
>> [ipv-rhel73:10575] mca: base: components_open: found loaded component tcp
>> [ipv-rhel73:10575] mca: base: components_open: component tcp open function 
>> successful
>> [ipv-rhel73:10575] mca:oob:select: checking available component tcp
>> [ipv-rhel73:10575] mca:oob:select: Querying component [tcp]
>> [ipv-rhel73:10575] oob:tcp: component_available called
>> [ipv-rhel73:10575] WORKING INTERFACE 1 KERNEL INDEX 2 FAMILY: V6
>> [ipv-rhel73:10575] [[20058,0],0] oob:tcp:init adding 
>> fe80::b9b:ac5d:9cf0:b858 to our list of V6 connections
>> [ipv-rhel73:10575] WORKING INTERFACE 2 KERNEL INDEX 1 FAMILY: V4
>> [ipv-rhel73:10575] [[20058,0],0] oob:tcp:init rejecting loopback interface lo
>> [ipv-rhel73:10575] WORKING INTERFACE 3 KERNEL INDEX 4 FAMILY: V4
>> [ipv-rhel73:10575] [[20058,0],0] TCP STARTUP
>> [ipv-rhel73:10575] [[20058,0],0] attempting to bind to IPv4 port 0
>> [ipv-rhel73:10575] [[20058,0],0] assigned IPv4 port 53438
>> [ipv-rhel73:10575] [[20058,0],0] attempting to bind to IPv6 port 0
>> [ipv-rhel73:10575] [[20058,0],0] assigned IPv6 port 43370
>> [ipv-rhel73:10575] mca:oob:select: Adding component to end
>> [ipv-rhel73:10575] mca:oob:select: Found 1 active transports
>> [ipv-rhel73:10575] [[20058,0],0]: get transports
>> [ipv-rhel73:10575] [[20058,0],0]:get transports for component tcp
>> [ipv-rhel73:10575] mca_base_component_repository_open: unable to open 
>> mca_ras_tm: libtorque.so.2: cannot open shared object file: No such file or 
>> directory (ignored)
>> [ipv-rhel71a.locallab.local:12299] mca: base: components_register: 
>> registering framework oob components
>> [ipv-rhel71a.locallab.local:12299] mca: base: components_register: found 
>> loaded component tcp
>> [ipv-rhel71a.locallab.local:12299] mca: base: components_register: component 
>> tcp register function successful
>> [ipv-rhel71a.locallab.local:12299] mca: base: components_open: opening oob 
>> components
>> [ipv-rhel71a.locallab.local:12299] mca: base: components_open: found loaded 
>> component tcp
>> [ipv-rhel71a.locallab.local:12299] mca: base: components_open: component tcp 
>> open function successful
>> [ipv-rhel71a.locallab.local:12299] mca:oob:select: checking available 
>> component tcp
>> [ipv-rhel71a.locallab.local:12299] mca:oob:select: Querying component [tcp]
>> [ipv-rhel71a.locallab.local:12299] oob:tcp: component_available called
>> [ipv-rhel71a.locallab.local:12299] WORKING INTERFACE 1 KERNEL INDEX 2 
>> FAMILY: V6
>> [ipv-rhel71a.locallab.local:12299] [[20058,0],1] oob:tcp:init adding 
>> fe80::226:b9ff:fe85:6a28 to our list of V6 connections
>> [ipv-rhel71a.locallab.local:12299] WORKING INTERFACE 2 KERNEL INDEX 1 
>

Re: [OMPI users] IpV6 Openmpi mpirun failed

[r...@open-mpi.org]

Looks like there is a firewall or something blocking communication between 
those nodes?

> On Oct 18, 2017, at 1:29 PM, Mukkie  wrote:
> 
> Adding a verbose output. Please check for failed and advise. Thank you.
> 
> [mselvam@ipv-rhel73 examples]$ mpirun -hostfile host --mca oob_base_verbose 
> 100 --mca btl tcp,self ring_c
> [ipv-rhel73:10575] mca_base_component_repository_open: unable to open 
> mca_plm_tm: libtorque.so.2: cannot open shared object file: No such file or 
> directory (ignored)
> [ipv-rhel73:10575] mca: base: components_register: registering framework oob 
> components
> [ipv-rhel73:10575] mca: base: components_register: found loaded component tcp
> [ipv-rhel73:10575] mca: base: components_register: component tcp register 
> function successful
> [ipv-rhel73:10575] mca: base: components_open: opening oob components
> [ipv-rhel73:10575] mca: base: components_open: found loaded component tcp
> [ipv-rhel73:10575] mca: base: components_open: component tcp open function 
> successful
> [ipv-rhel73:10575] mca:oob:select: checking available component tcp
> [ipv-rhel73:10575] mca:oob:select: Querying component [tcp]
> [ipv-rhel73:10575] oob:tcp: component_available called
> [ipv-rhel73:10575] WORKING INTERFACE 1 KERNEL INDEX 2 FAMILY: V6
> [ipv-rhel73:10575] [[20058,0],0] oob:tcp:init adding fe80::b9b:ac5d:9cf0:b858 
> to our list of V6 connections
> [ipv-rhel73:10575] WORKING INTERFACE 2 KERNEL INDEX 1 FAMILY: V4
> [ipv-rhel73:10575] [[20058,0],0] oob:tcp:init rejecting loopback interface lo
> [ipv-rhel73:10575] WORKING INTERFACE 3 KERNEL INDEX 4 FAMILY: V4
> [ipv-rhel73:10575] [[20058,0],0] TCP STARTUP
> [ipv-rhel73:10575] [[20058,0],0] attempting to bind to IPv4 port 0
> [ipv-rhel73:10575] [[20058,0],0] assigned IPv4 port 53438
> [ipv-rhel73:10575] [[20058,0],0] attempting to bind to IPv6 port 0
> [ipv-rhel73:10575] [[20058,0],0] assigned IPv6 port 43370
> [ipv-rhel73:10575] mca:oob:select: Adding component to end
> [ipv-rhel73:10575] mca:oob:select: Found 1 active transports
> [ipv-rhel73:10575] [[20058,0],0]: get transports
> [ipv-rhel73:10575] [[20058,0],0]:get transports for component tcp
> [ipv-rhel73:10575] mca_base_component_repository_open: unable to open 
> mca_ras_tm: libtorque.so.2: cannot open shared object file: No such file or 
> directory (ignored)
> [ipv-rhel71a.locallab.local:12299] mca: base: components_register: 
> registering framework oob components
> [ipv-rhel71a.locallab.local:12299] mca: base: components_register: found 
> loaded component tcp
> [ipv-rhel71a.locallab.local:12299] mca: base: components_register: component 
> tcp register function successful
> [ipv-rhel71a.locallab.local:12299] mca: base: components_open: opening oob 
> components
> [ipv-rhel71a.locallab.local:12299] mca: base: components_open: found loaded 
> component tcp
> [ipv-rhel71a.locallab.local:12299] mca: base: components_open: component tcp 
> open function successful
> [ipv-rhel71a.locallab.local:12299] mca:oob:select: checking available 
> component tcp
> [ipv-rhel71a.locallab.local:12299] mca:oob:select: Querying component [tcp]
> [ipv-rhel71a.locallab.local:12299] oob:tcp: component_available called
> [ipv-rhel71a.locallab.local:12299] WORKING INTERFACE 1 KERNEL INDEX 2 FAMILY: 
> V6
> [ipv-rhel71a.locallab.local:12299] [[20058,0],1] oob:tcp:init adding 
> fe80::226:b9ff:fe85:6a28 to our list of V6 connections
> [ipv-rhel71a.locallab.local:12299] WORKING INTERFACE 2 KERNEL INDEX 1 FAMILY: 
> V4
> [ipv-rhel71a.locallab.local:12299] [[20058,0],1] oob:tcp:init rejecting 
> loopback interface lo
> [ipv-rhel71a.locallab.local:12299] [[20058,0],1] TCP STARTUP
> [ipv-rhel71a.locallab.local:12299] [[20058,0],1] attempting to bind to IPv4 
> port 0
> [ipv-rhel71a.locallab.local:12299] [[20058,0],1] assigned IPv4 port 50782
> [ipv-rhel71a.locallab.local:12299] [[20058,0],1] attempting to bind to IPv6 
> port 0
> [ipv-rhel71a.locallab.local:12299] [[20058,0],1] assigned IPv6 port 59268
> [ipv-rhel71a.locallab.local:12299] mca:oob:select: Adding component to end
> [ipv-rhel71a.locallab.local:12299] mca:oob:select: Found 1 active transports
> [ipv-rhel71a.locallab.local:12299] [[20058,0],1]: get transports
> [ipv-rhel71a.locallab.local:12299] [[20058,0],1]:get transports for component 
> tcp
> [ipv-rhel71a.locallab.local:12299] [[20058,0],1]: set_addr to uri 
> 1314521088.0;tcp6://[fe80::b9b:ac5d:9cf0:b858]:43370
> [ipv-rhel71a.locallab.local:12299] [[20058,0],1]:set_addr checking if peer 
> [[20058,0],0] is reachable via component tcp
> [ipv-rhel71a.locallab.local:12299] [[20058,0],1] oob:tcp: working peer 
> [[20058,0],0] address tcp6://[fe80::b9b:ac5d:9cf0:b858]:43370
> [ipv-rhel71a.locallab.local:12299] [[20058,0],1] SET_PEER ADDING PEER 
> [[20058,0],0]
> [ipv-rhel71a.locallab.local:12299] [[20058,0],1] set_peer: peer [[20058,0],0] 
> is listening on net fe80::b9b:ac5d:9cf0:b858 port 43370
> [ipv-rhel71a.locallab.local:12299] [[20058,0],1]: peer [[20058,0],0] is 
> reachable 

Re: [OMPI users] Failed to register memory (openmpi 2.0.2)

[r...@open-mpi.org]

Put “oob=tcp” in your default MCA param file

> On Oct 18, 2017, at 9:00 AM, Mark Dixon  wrote:
> 
> Hi,
> 
> We're intermittently seeing messages (below) about failing to register memory 
> with openmpi 2.0.2 on centos7 / Mellanox FDR Connect-X 3 and the vanilla IB 
> stack as shipped by centos.
> 
> We're not using any mlx4_core module tweaks at the moment. On earlier 
> machines we used to set registered memory as per the FAQ, but neither 
> log_num_mtt nor num_mtt seem to exist these days (according to 
> /sys/module/mlx4_*/parameters/*), which makes it somewhat difficult to follow 
> the FAQ.
> 
> The output of 'ulimit -l' shows as unlimited for every rank.
> 
> Does anyone have any advice, please?
> 
> Thanks,
> 
> Mark
> 
> -
> Failed to register memory region (MR):
> 
> Hostname: dc1s0b1c
> Address:  ec5000
> Length:   20480
> Error:Cannot allocate memory
> --
> --
> Open MPI has detected that there are UD-capable Verbs devices on your
> system, but none of them were able to be setup properly.  This may
> indicate a problem on this system.
> 
> You job will continue, but Open MPI will ignore the "ud" oob component
> in this run.
> ___
> users mailing list
> users@lists.open-mpi.org
> https://lists.open-mpi.org/mailman/listinfo/users

___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users

Re: [OMPI users] Controlling spawned process

[r...@open-mpi.org]

Couple of things you can try:

* add --oversubscribe to your mpirun cmd line so it doesn’t care how many slots 
there are

* modify your MPI_INFO to be “host”, “node0:22” so it thinks there are more 
slots available

It’s possible that the “host” info processing has a bug in it, but this will 
tell us a little more and hopefully get your running. If you want to bind your 
processes to cores, then add “--bind-to core” to the cmd line



> On Oct 6, 2017, at 1:35 PM, George Reeke  wrote:
> 
> Dear colleagues,
> I need some help controlling where a process spawned with
> MPI_Comm_spawn goes.  I am in openmpi-1.10 under Centos 6.7.
> My application is written in C and am running on a RedBarn
> system with a master node (hardware box) that connects to the
> outside world and two other nodes connected to it via ethernet and
> Infiniband.  There are two executable files, one (I'll call it
> "Rank0Pgm") that expects to be rank 0 and does all the I/O and
> the other ("RanknPgm") that only communicates via MPI messages.
> There are two MPI_Comm_spawns that run just after MPI_Init and
> an initial broadcast that shares some setup info, like this:
> MPI_Comm_spawn("andmsg", argv, 1, MPI_INFO_NULL,
>   hostid, commc, , );
> where "andmsg" is a program that needs to communicate with the
> internet and with all the other processes via a new communicator
> that will be called commd (and another name for the other one).
>   When I run this program with no hostfile and an mpirun line
> something like this on a node with 32 cores:
> /usr/lib64/openmpi-1.10/bin/mpirun -n 1 Rank0Pgm : -n 28 RanknPgm \
>   < InputFile
> everything works fine.  I assume the spawns use 2 of the 3 available
> cores that I did not ask the program to use.
> 
> Now I want to run on the full network, so I make a hostfile like this
> (call it "nodes120"):
> node0 slots=22 max-slots=22
> n0003 slots=40 max-slots=40
> n0004 slots=56 max-slots=56
> where node0 has 24 cores and I am trying to leave room for my two
> spawned processes.  The spawned processes have to be able to contact
> the internet, so I make an MPI_INFO with MPI_Info_create and
> MPI_Info_set(mpinfo, "host", "node0")
> and change the MPI_INFO_NULL in the spawn calls to point to this
> new MPI_Info.  (If I leave the MPI_INFO_NULL I get a different
> error that is probably not of interest here.)
> 
> Now I run the mpirun like above except now with
> "--hostfile nodes120" and "-n 116" after the colon.  Now I get this
> error:
> 
> "There are not enough slots available in the system to satisfy the 1
> slots that were requested by the application:
>  andmsg
> Either request fewer slots for your application, or make more slots
> available for use."
> 
> I get the same error with "max-slots=24" on the first line of the
> hosts file.
> 
> Sorry for the length of all that.  Request for help:  How do I set
> things up to run my rank 0 program and enough copies of RanknPgm to fill
> all but some number of cores on the master hardware node, and all the
> other rank n programs on the other hardware "nodes" (boxes of CPUs).
> [My application will do best with the default "by slot" scheduling.]
> 
> Suggestions much appreciated.  I am quite convinced my code is OK
> in that it runs OK as shown above on one hardware box.  Also runs
> on my laptop with 4 cores and "-n 3 RanknPgm" so I guess I don't
> even really need to reserve cores for the two spawned processes.
> I thought of using old-fashioned 'fork' but I really want the
> extra communicators to keep asynchronous messages separated.
> The documentation says overloading is OK by default, so maybe
> something else is wrong here.
> 
> George Reeke
> 
> 
> 
> 
> ___
> users mailing list
> users@lists.open-mpi.org
> https://lists.open-mpi.org/mailman/listinfo/users

___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users

Re: [OMPI users] OpenMPI with-tm is not obeying torque

[r...@open-mpi.org]

No problem - glad you were able to work it out!


> On Oct 5, 2017, at 11:22 PM, Anthony Thyssen <a.thys...@griffith.edu.au> 
> wrote:
> 
> Sorry  r...@open-mpi.org <mailto:r...@open-mpi.org>  as Gilles Gouaillardet 
> pointed out to me the problem wasn't OpenMPI, but with the specific EPEL 
> implementation (see Redhat Bugzilla 1321154)
> 
> Today, the the server was able to be taken down for maintance, and I wanted 
> to try a few things.
> 
> After installing EPEL Testing Repotorque-4.2.10-11.el7
> 
> However I found that all the nodes were 'down'  even though everything 
> appears to be running, with no errors in the error logs.
> 
> After a lot of trials, errors and reseach, I eventually (on a whim) I decided 
> to remove the "num_node_boards=1" entry from the "torque/server_priv/nodes" 
> file and restart the server & scheduler.   Suddenly the nodes were "free" and 
> my initial test job ran.
> 
> Perhaps the EPEL-Test Torque 4.2.10-11  does not contain Numa?
> 
> ALL later tests (with OpenMPI - RHEL SRPM 1.10.6-2 re-compiled "--with-tm")  
> is now responding to the Torque mode allocation correctly and is no longer 
> simply running all the jobs on the first node.
> 
> That is$PBS_NODEFILE  ,pbsdsh hostname  and   mpirun hostnameare 
> all in agreement.
> 
> Thank you all for your help, and putting up with with me.
> 
>   Anthony Thyssen ( System Programmer )<a.thys...@griffith.edu.au 
> <mailto:a.thys...@griffith.edu.au>>
>  --
>   "Around here we've got a name for people what talks to dragons."
>   "Traitor?"  Wiz asked apprehensively.
>   "No.  Lunch."     -- Rick Cook, "Wizadry Consulted"
>  --
> 
> 
> On Wed, Oct 4, 2017 at 11:43 AM, r...@open-mpi.org <mailto:r...@open-mpi.org> 
> <r...@open-mpi.org <mailto:r...@open-mpi.org>> wrote:
> Can you try a newer version of OMPI, say the 3.0.0 release? Just curious to 
> know if we perhaps “fixed” something relevant.
> 
> 
>> On Oct 3, 2017, at 5:33 PM, Anthony Thyssen <a.thys...@griffith.edu.au 
>> <mailto:a.thys...@griffith.edu.au>> wrote:
>> 
>> FYI...
>> 
>> The problem is discussed further in 
>> 
>> Redhat Bugzilla: Bug 1321154 - numa enabled torque don't work
>>https://bugzilla.redhat.com/show_bug.cgi?id=1321154 
>> <https://bugzilla.redhat.com/show_bug.cgi?id=1321154>
>> 
>> I'd seen this previous as it required me to add "num_node_boards=1" to each 
>> node in the
>> /var/lib/torque/server_priv/nodes  to get torque to at least work.  
>> Specifically by munging
>> the $PBS_NODES" (which comes out correcT) into a host list containing the 
>> correct
>> "slot=" counts.  But of course now that I have compiled OpenMPI using 
>> "--with-tm" that
>> should not have been needed as in fact is now ignored by OpenMPI in a 
>> Torque-PBS
>> environment.
>> 
>> However it seems ever since "NUMA" support was into the Torque RPM's, has 
>> also caused
>> the current problems, and is still continuing.   The last action is a new 
>> EPEL "test' version
>> (August 2017),  I will try shortly.
>> 
>> Take you for your help, though I am still open to suggestions for a 
>> replacement.
>> 
>>   Anthony Thyssen ( System Programmer )<a.thys...@griffith.edu.au 
>> <mailto:a.thys...@griffith.edu.au>>
>>  --
>>Encryption... is a powerful defensive weapon for free people.
>>It offers a technical guarantee of privacy, regardless of who is
>>running the government... It's hard to think of a more powerful,
>>less dangerous tool for liberty.--  Esther Dyson
>>  --
>> 
>> 
>> 
>> On Wed, Oct 4, 2017 at 9:02 AM, Anthony Thyssen <a.thys...@griffith.edu.au 
>> <mailto:a.thys...@griffith.edu.au>> wrote:
>> Thank you Gilles.  At least I now have something to follow though with.
>> 
>> As a FYI, the torque is the pre-built version from the Redhat Extras (EPEL) 
>> archive.
>> torque-4.2.10-10.el7.x86_64
>> 
>> Normally pre-build packages have no problems, but in this case.
>> 
>> 
>> 
>> 
>> On Tue, Oct 3, 2017 at 3:39 PM, Gilles Gouail

Re: [OMPI users] OpenMPI with-tm is not obeying torque

[r...@open-mpi.org]

Can you try a newer version of OMPI, say the 3.0.0 release? Just curious to 
know if we perhaps “fixed” something relevant.


> On Oct 3, 2017, at 5:33 PM, Anthony Thyssen  wrote:
> 
> FYI...
> 
> The problem is discussed further in 
> 
> Redhat Bugzilla: Bug 1321154 - numa enabled torque don't work
>https://bugzilla.redhat.com/show_bug.cgi?id=1321154 
> 
> 
> I'd seen this previous as it required me to add "num_node_boards=1" to each 
> node in the
> /var/lib/torque/server_priv/nodes  to get torque to at least work.  
> Specifically by munging
> the $PBS_NODES" (which comes out correcT) into a host list containing the 
> correct
> "slot=" counts.  But of course now that I have compiled OpenMPI using 
> "--with-tm" that
> should not have been needed as in fact is now ignored by OpenMPI in a 
> Torque-PBS
> environment.
> 
> However it seems ever since "NUMA" support was into the Torque RPM's, has 
> also caused
> the current problems, and is still continuing.   The last action is a new 
> EPEL "test' version
> (August 2017),  I will try shortly.
> 
> Take you for your help, though I am still open to suggestions for a 
> replacement.
> 
>   Anthony Thyssen ( System Programmer ) >
>  --
>Encryption... is a powerful defensive weapon for free people.
>It offers a technical guarantee of privacy, regardless of who is
>running the government... It's hard to think of a more powerful,
>less dangerous tool for liberty.--  Esther Dyson
>  --
> 
> 
> 
> On Wed, Oct 4, 2017 at 9:02 AM, Anthony Thyssen  > wrote:
> Thank you Gilles.  At least I now have something to follow though with.
> 
> As a FYI, the torque is the pre-built version from the Redhat Extras (EPEL) 
> archive.
> torque-4.2.10-10.el7.x86_64
> 
> Normally pre-build packages have no problems, but in this case.
> 
> 
> 
> 
> On Tue, Oct 3, 2017 at 3:39 PM, Gilles Gouaillardet  > wrote:
> Anthony,
> 
> 
> we had a similar issue reported some times ago (e.g. Open MPI ignores torque 
> allocation),
> 
> and after quite some troubleshooting, we ended up with the same behavior 
> (e.g. pbsdsh is not working as expected).
> 
> see https://www.mail-archive.com/users@lists.open-mpi.org/msg29952.html 
>  for the 
> last email.
> 
> 
> from an Open MPI point of view, i would consider the root cause is with your 
> torque install.
> 
> this case was reported at 
> http://www.clusterresources.com/pipermail/torqueusers/2016-September/018858.html
>  
> 
> 
> and no conclusion was reached.
> 
> 
> Cheers,
> 
> 
> Gilles
> 
> 
> On 10/3/2017 2:02 PM, Anthony Thyssen wrote:
> The stdin and stdout are saved to separate channels.
> 
> It is interesting that the output from pbsdsh is node21.emperor 5 times, even 
> though $PBS_NODES is the 5 individual nodes.
> 
> Attached are the two compressed files, as well as the pbs_hello batch used.
> 
> Anthony Thyssen ( System Programmer )   >>
>  --
>   There are two types of encryption:
> One that will prevent your sister from reading your diary, and
> One that will prevent your government.   -- Bruce Schneier
>  --
> 
> 
> 
> 
> On Tue, Oct 3, 2017 at 2:39 PM, Gilles Gouaillardet    >> wrote:
> 
> Anthony,
> 
> 
> in your script, can you
> 
> 
> set -x
> 
> env
> 
> pbsdsh hostname
> 
> mpirun --display-map --display-allocation --mca ess_base_verbose
> 10 --mca plm_base_verbose 10 --mca ras_base_verbose 10 hostname
> 
> 
> and then compress and send the output ?
> 
> 
> Cheers,
> 
> 
> Gilles
> 
> 
> On 10/3/2017 1:19 PM, Anthony Thyssen wrote:
> 
> I noticed that too.  Though the submitting host for torque is
> a different host (main head node, "shrek"),  "node21" is the
> host that torque runs the batch script (and the mpirun
> command) it being the first node in the "dualcore" resource group.
> 
> Adding option...
> 
> It fixed the hostname in the allocation map, though had no
> effect on the outcome.  The allocation is still simply ignored.
> 
> 

Re: [OMPI users] OMPI users] upgraded mpi and R and now cannot find slots

[r...@open-mpi.org]

You can add it to the default MCA param file, if you want - 
/etc/openmpi-mca-params.conf

> On Oct 3, 2017, at 12:44 PM, Jim Maas <jimmaa...@gmail.com> wrote:
> 
> Thanks RHC  where do I put that so it will be in the environment?
> 
> J
> 
> On 3 October 2017 at 16:01, r...@open-mpi.org <mailto:r...@open-mpi.org> 
> <r...@open-mpi.org <mailto:r...@open-mpi.org>> wrote:
> As Gilles said, we default to slots = cores, not HTs. If you want to treat 
> HTs as independent cpus, then you need to add 
> OMPI_MCA_hwloc_base_use_hwthreads_as_cpus=1 in your environment.
> 
>> On Oct 3, 2017, at 7:27 AM, Jim Maas <jimmaa...@gmail.com 
>> <mailto:jimmaa...@gmail.com>> wrote:
>> 
>> Tried this and got this error, and slots are available, nothing else is 
>> running.
>> 
>> > cl <- startMPIcluster(count=7)
>> --
>> There are not enough slots available in the system to satisfy the 7 slots
>> that were requested by the application:
>>   /usr/local/lib/R/bin/Rscript
>> 
>> Either request fewer slots for your application, or make more slots available
>> for use.
>> --
>> Error in mpi.comm.spawn(slave = rscript, slavearg = args, nslaves = count,  
>> : 
>>   MPI_ERR_SPAWN: could not spawn processes
>> > 
>> 
>> On 3 October 2017 at 15:07, Gilles Gouaillardet 
>> <gilles.gouaillar...@gmail.com <mailto:gilles.gouaillar...@gmail.com>> wrote:
>> Thanks, i will have a look at it.
>> 
>> By default, a slot is a core, so there are 6 slots on your system.
>> Could your app spawn 6 procs on top of the initial proc ? That would be 7 
>> slots and there are only 6.
>> What if you ask 5 slots only ?
>> 
>> With some parameters i do not know off hand, you could either oversubscribe 
>> or use hyperthreads as slots. In both cases, 7 slots would be available.
>> 
>> Cheers,
>> 
>> Gilles
>> 
>> Jim Maas <jimmaa...@gmail.com <mailto:jimmaa...@gmail.com>> wrote:
>> Thanks Gilles, relative noob here at this level, apologies if nonsensical!
>> 
>> I removed previous versions of open mpi which were compiled from source 
>> using sudo make uninstall ...
>> downloaded new open-mpi 3.0.0 in tar.gz
>> configure --disable-dlopen
>> sudo make install
>> 
>> 
>> then ran sudo ldconfig
>> 
>> updated R, downloaded R-3.4.2.tar.gz
>> ./configure
>> sudo make install
>> 
>> 
>> Then run R from sudo
>> 
>> sudo R
>> once running 
>> install.packages("Rmpi")
>> install.packages("doMPI")
>> 
>> both of these load and test fine during install
>> 
>> Then from R run
>> 
>> rm(list=ls(all=TRUE))
>> library(doMPI)
>> 
>> ## load MPI cluster
>> cl <- startMPIcluster(count=6)
>> 
>> 
>> At this point it throws the error, doesn't find any of the slots.
>> 
>> There is a precompiled version of Rmpi that installs an older version of 
>> open-mpi directly from Ubuntu, but I think the mpi version is an older one 
>> so I wanted to try using the new version.
>> 
>> 
>> I use this 6 core (12) as  test bed before uploading to a cluster.  It is 
>> Ubuntu 16.04 Linux, lstopo pdf is attached.
>> 
>> Thanks,
>> 
>> J
>> 
>> 
>> On 3 October 2017 at 14:06, Gilles Gouaillardet 
>> <gilles.gouaillar...@gmail.com <mailto:gilles.gouaillar...@gmail.com>> wrote:
>> Hi Jim,
>> 
>> can you please provide minimal instructions on how to reproduce the issue ?
>> we know Open MPI, but i am afraid few or none of us know about Rmpi nor 
>> doMPI.
>> once you explain how to download and build these, and how to run the
>> failing test,
>> we ll be able to investigate that.
>> 
>> also, can you describe your environment ?
>> i assume one ubuntu machine, can you please run
>> lstopo
>> on and post the output ?
>> 
>> did you use to have some specific settings in the system-wide conf
>> file (e.g. /.../etc/openmpi-mca-params.co <http://openmpi-mca-params.co/>nf) 
>> ?
>> if yes, can you post these, the syntax might have changed in 3.0.0
>> 
>> Cheers,
>> 
>> Gilles
>> 
>> On Tue, Oct 3, 2017 at 7:34 PM, Jim Maas <jimmaa...@gmail.com 
>> <mailto:jimmaa...@gmail.com>> wrote:
>> > I've used this for years, just updated o

Re: [OMPI users] OMPI users] upgraded mpi and R and now cannot find slots

[r...@open-mpi.org]

As Gilles said, we default to slots = cores, not HTs. If you want to treat HTs 
as independent cpus, then you need to add 
OMPI_MCA_hwloc_base_use_hwthreads_as_cpus=1 in your environment.

> On Oct 3, 2017, at 7:27 AM, Jim Maas  wrote:
> 
> Tried this and got this error, and slots are available, nothing else is 
> running.
> 
> > cl <- startMPIcluster(count=7)
> --
> There are not enough slots available in the system to satisfy the 7 slots
> that were requested by the application:
>   /usr/local/lib/R/bin/Rscript
> 
> Either request fewer slots for your application, or make more slots available
> for use.
> --
> Error in mpi.comm.spawn(slave = rscript, slavearg = args, nslaves = count,  : 
>   MPI_ERR_SPAWN: could not spawn processes
> > 
> 
> On 3 October 2017 at 15:07, Gilles Gouaillardet 
> > wrote:
> Thanks, i will have a look at it.
> 
> By default, a slot is a core, so there are 6 slots on your system.
> Could your app spawn 6 procs on top of the initial proc ? That would be 7 
> slots and there are only 6.
> What if you ask 5 slots only ?
> 
> With some parameters i do not know off hand, you could either oversubscribe 
> or use hyperthreads as slots. In both cases, 7 slots would be available.
> 
> Cheers,
> 
> Gilles
> 
> Jim Maas > wrote:
> Thanks Gilles, relative noob here at this level, apologies if nonsensical!
> 
> I removed previous versions of open mpi which were compiled from source using 
> sudo make uninstall ...
> downloaded new open-mpi 3.0.0 in tar.gz
> configure --disable-dlopen
> sudo make install
> 
> 
> then ran sudo ldconfig
> 
> updated R, downloaded R-3.4.2.tar.gz
> ./configure
> sudo make install
> 
> 
> Then run R from sudo
> 
> sudo R
> once running 
> install.packages("Rmpi")
> install.packages("doMPI")
> 
> both of these load and test fine during install
> 
> Then from R run
> 
> rm(list=ls(all=TRUE))
> library(doMPI)
> 
> ## load MPI cluster
> cl <- startMPIcluster(count=6)
> 
> 
> At this point it throws the error, doesn't find any of the slots.
> 
> There is a precompiled version of Rmpi that installs an older version of 
> open-mpi directly from Ubuntu, but I think the mpi version is an older one so 
> I wanted to try using the new version.
> 
> 
> I use this 6 core (12) as  test bed before uploading to a cluster.  It is 
> Ubuntu 16.04 Linux, lstopo pdf is attached.
> 
> Thanks,
> 
> J
> 
> 
> On 3 October 2017 at 14:06, Gilles Gouaillardet 
> > wrote:
> Hi Jim,
> 
> can you please provide minimal instructions on how to reproduce the issue ?
> we know Open MPI, but i am afraid few or none of us know about Rmpi nor doMPI.
> once you explain how to download and build these, and how to run the
> failing test,
> we ll be able to investigate that.
> 
> also, can you describe your environment ?
> i assume one ubuntu machine, can you please run
> lstopo
> on and post the output ?
> 
> did you use to have some specific settings in the system-wide conf
> file (e.g. /.../etc/openmpi-mca-params.co nf) ?
> if yes, can you post these, the syntax might have changed in 3.0.0
> 
> Cheers,
> 
> Gilles
> 
> On Tue, Oct 3, 2017 at 7:34 PM, Jim Maas  > wrote:
> > I've used this for years, just updated open-mpi to 3.0.0 and reloaded R,
> > have reinstalled doMPI and thus Rmpi but when I try to use startMPICluster,
> > asking for 6 slots (there are 12 on this machine) I get this error.  Where
> > can I start to debug it?
> >
> > Thanks
> > J
> > --
> > There are not enough slots available in the system to satisfy the 6 slots
> > that were requested by the application:
> >   /usr/lib/R/bin/Rscript
> >
> > Either request fewer slots for your application, or make more slots
> > available
> > for use.
> > --
> > Error in mpi.comm.spawn(slave = rscript, slavearg = args, nslaves = count,
> > :
> >   MPI_ERR_SPAWN: could not spawn processes
> > --
> > Jim Maas
> >
> > jimmaasuk  at gmail.com 
> >
> >
> > ___
> > users mailing list
> > users@lists.open-mpi.org 
> > https://lists.open-mpi.org/mailman/listinfo/users 
> > 
> ___
> users mailing list
> users@lists.open-mpi.org 
> https://lists.open-mpi.org/mailman/listinfo/users 
> 
> 
> 
> 
> -- 
> 
> 

Re: [OMPI users] OpenMPI with-tm is not obeying torque

[r...@open-mpi.org]

One thing I can see is that the local host (where mpirun executed) shows as 
“node21” in the allocation, while all others show their FQDN. This might be 
causing some confusion.

You might try adding "--mca orte_keep_fqdn_hostnames 1” to your cmd line and 
see if that helps.


> On Oct 2, 2017, at 8:14 PM, Anthony Thyssen  wrote:
> 
> Update...  Problem of all processes runing on first node (oversubscribed 
> dual-core machine) is NOT resolved.
> 
> Changing the mpirun  command in the Torque batch script ("pbs_hello" - See 
> previous) to
> 
>mpirun --nooversubscribe --display-allocation hostname
> 
> Then submitting to PBS/Torque using
> 
>   qsub -l nodes=5:ppn=1:dualcore pbs_hello
> 
> To run on 5 dual-core machines.  Produces the following result...
> 
> ===8 PBS Job Number   8996
> PBS batch run on node21.emperor
> Time it was started  2017-10-03_13:04:07
> Current Directory/net/shrek.emperor/home/shrek/anthony
> Submitted work dir   /home/shrek/anthony/mpi-pbs
> Number of Nodes  5
> Nodefile List/var/lib/torque/aux//8996.shrek.emperor
> node21.emperor
> node25.emperor
> node24.emperor
> node23.emperor
> node22.emperor
> ---
> 
> ==   ALLOCATED NODES   ==
> node21: slots=1 max_slots=0 slots_inuse=0 state=UP
> node25.emperor: slots=1 max_slots=0 slots_inuse=0 state=UP
> node24.emperor: slots=1 max_slots=0 slots_inuse=0 state=UP
> node23.emperor: slots=1 max_slots=0 slots_inuse=0 state=UP
> node22.emperor: slots=1 max_slots=0 slots_inuse=0 state=UP
> =
> node21.emperor
> node21.emperor
> node21.emperor
> node21.emperor
> node21.emperor
> ===8 
> The $PBS_NODE file shows torque requesting 5 processes on 5 separate machines.
> 
> The mpirun command's "ALLOCATED NODES" shows it picked up the request 
> correctly from torque.
> 
> But the "hostname" output still shows ALL processes were run on the first 
> node only!
> 
> Even though I requested it not to over subscribe.
> 
> 
> I am at a complete loss as to how to solve this problem..
> 
> ANY and all suggestions, or even ways I can get other information as to what 
> is causing this will be most welcome.
> 
> 
>   Anthony Thyssen ( System Programmer ) >
>  --
>Using encryption on the Internet is the equivalent of arranging 
>an armored car to deliver credit-card information from someone 
>living in a cardboard box to someone living on a park bench.   
>-- Gene Spafford
>  --
> 
> 
> 
> ___
> users mailing list
> users@lists.open-mpi.org
> https://lists.open-mpi.org/mailman/listinfo/users

___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users

Re: [OMPI users] OpenMPI 3.0.0, compilation using Intel icc 11.1 on Linux, error when compiling pmix_mmap

[r...@open-mpi.org]

I correctly understood the file and the errors. I’m just pointing out that the 
referenced file cannot possibly contain a pointer to opal/threads/condition.h. 
There is no include in that file that can pull in that path.


> On Oct 2, 2017, at 11:39 AM, Jeff Squyres (jsquyres) <jsquy...@cisco.com> 
> wrote:
> 
> Ralph --
> 
> I think he cited a typo in his email.  The actual file he is referring to is 
> 
> -
> $ find . -name pmix_mmap.c
> ./opal/mca/pmix/pmix2x/pmix/src/sm/pmix_mmap.c
> -
> 
> From his log file, there appear to be two problems:
> 
> -
> sm/pmix_mmap.c(66): warning #266: function "posix_fallocate" declared 
> implicitly
>  if (0 != (rc = posix_fallocate(sm_seg->seg_id, 0, size))) {
> ^
> 
> sm/pmix_mmap.c(88): warning #266: function "ftruncate" declared implicitly
>  if (0 != ftruncate(sm_seg->seg_id, size)) {
>   ^
> -
> 
> (which are just warnings but we should probably fix them)
> 
> and
> 
> -
> /opt/openmpi-3.0.0-Intel-src/opal/threads/condition.h(96): error: pointer to 
> incomplete class type is not allowed
>  absolute.tv_sec = abstime->tv_sec;
>^
> -
> 
> This one appears to be the actual error.
> 
> abstime is a (struct timespec), which, according to 
> http://pubs.opengroup.org/onlinepubs/7908799/xsh/time.h.html, should be 
> declared in , which is definitely #included by 
> opal/threads/condition.h.
> 
> Since this error occurred with Intel 11.x but didn't occur with later 
> versions of the Intel compiler, I'm wondering if the Intel 11.x compiler 
> suite didn't support (struct timespec).
> 
> Can you stop using Intel 11.x and only use later versions of the Intel 
> compiler?
> 
> 
> 
>> On Oct 1, 2017, at 11:59 PM, r...@open-mpi.org wrote:
>> 
>> Afraid I’m rather stumped on this one. There is no such include file in 
>> pmix_mmap, nor is there any include file that might lead to it. You might 
>> try starting again from scratch to ensure you aren’t getting some weird 
>> artifact.
>> 
>> 
>>> On Sep 29, 2017, at 1:12 PM, Ted Sussman <ted.suss...@adina.com> wrote:
>>> 
>>> Hello all,
>>> 
>>> I downloaded openmpi-3.0.0.tar.gz and attempted to build it on my Red Hat 
>>> Linux computer, kernel 2.6.18-194.el5.
>>> 
>>> The C compiler used is Intel icc, version 11.1.
>>> 
>>> The make failed when compiling pmix_mmap, with messages such as
>>> 
>>> /opt/openmpi-3.0.0-Intel-src/opal/threads/conditions.h(96): error: pointer 
>>> to incomplete class type is not allowed
>>> 
>>>absolute.tv_sec = abstime->tv_sec;
>>> 
>>> I have attached a tar file with the output from configure and the output 
>>> from make.
>>> 
>>> I was able to build openmpi-2.1.1 using the same computer and compiler.
>>> 
>>> I was able to build openmpi-3.0.0 using a different computer, with icc 
>>> version 14.0.4.
>>> 
>>> Can you please tell me how I can avoid this compilation error, when using 
>>> icc version 11.1?
>>> 
>>> Sincerely,
>>> 
>>> Ted Sussman
>>> 
>>> The following section of this message contains a file attachment
>>> prepared for transmission using the Internet MIME message format.
>>> If you are using Pegasus Mail, or any other MIME-compliant system,
>>> you should be able to save it or view it from within your mailer.
>>> If you cannot, please ask your system administrator for assistance.
>>> 
>>>   File information ---
>>>File:  openmpi.tgz
>>>Date:  29 Sep 2017, 15:59
>>>Size:  41347 bytes.
>>>Type:  Binary
>>> ___
>>> users mailing list
>>> users@lists.open-mpi.org
>>> https://lists.open-mpi.org/mailman/listinfo/users
>> 
>> ___
>> users mailing list
>> users@lists.open-mpi.org
>> https://lists.open-mpi.org/mailman/listinfo/users
> 
> 
> -- 
> Jeff Squyres
> jsquy...@cisco.com
> 
> ___
> users mailing list
> users@lists.open-mpi.org
> https://lists.open-mpi.org/mailman/listinfo/users

___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users

Re: [OMPI users] Fwd: OpenMPI does not obey hostfile

[r...@open-mpi.org]

That is correct. If you don’t specify a slot count, we auto-discover the number 
of cores on each node and set #slots to that number. If an RM is involved, then 
we use what they give us

Sent from my iPad

> On Sep 26, 2017, at 8:11 PM, Anthony Thyssen  
> wrote:
> 
> 
> I have been having problems with OpenMPI on a new cluster of machines, using
> stock RHEL7 packages.
> 
> ASIDE: This will be used with Torque-PBS (from EPEL archives), though OpenMPI
> (currently) does not have the "tm" resource manager configured to use PBS, as 
> you
> will be able to see in the debug output below.
> 
> # mpirun -V
> mpirun (Open MPI) 1.10.6
> 
> # sudo yum list installed openmpi
> ...
> Installed Packages
> openmpi.x86_641.10.6-2.el7@rhel-7-server-rpms
> ...
> 
> More than likely I am doing something fundamentally stupid, but I have no 
> idea what.
> 
> The problem is that OpenMPI is not obeying the given hostfile, and running one
> process on each host given in the list. The manual and all my (meagre) 
> experience
> is that that is what it is meant to do.
> 
> Instead it runs the maximum number of processes that is allowed to run for 
> the CPU
> of that machine.  That is a nice feature, but NOT what is wanted.
> 
> There is no "/etc/openmpi-x86_64/openmpi-default-hostfile" configuration 
> present.
> 
> For example given the hostfile
> 
> # cat hostfile.txt
> node21.emperor
> node22.emperor
> node22.emperor
> node23.emperor
> 
> Running OpenMPI on the head node "shrek", I get the following,
> (ras debugging enabled to see the result)
> 
> # mpirun --hostfile hostfile.txt --mca ras_base_verbose 5 mpi_hello
> [shrek.emperor:93385] mca:base:select:(  ras) Querying component [gridengine]
> [shrek.emperor:93385] mca:base:select:(  ras) Skipping component 
> [gridengine]. Query failed to return a module
> [shrek.emperor:93385] mca:base:select:(  ras) Querying component [loadleveler]
> [shrek.emperor:93385] mca:base:select:(  ras) Skipping component 
> [loadleveler]. Query failed to return a module
> [shrek.emperor:93385] mca:base:select:(  ras) Querying component [simulator]
> [shrek.emperor:93385] mca:base:select:(  ras) Skipping component [simulator]. 
> Query failed to return a module
> [shrek.emperor:93385] mca:base:select:(  ras) Querying component [slurm]
> [shrek.emperor:93385] mca:base:select:(  ras) Skipping component [slurm]. 
> Query failed to return a module
> [shrek.emperor:93385] mca:base:select:(  ras) No component selected!
> 
> ==   ALLOCATED NODES   ==
> node21.emperor: slots=1 max_slots=0 slots_inuse=0 state=UNKNOWN
> node22.emperor: slots=2 max_slots=0 slots_inuse=0 state=UNKNOWN
> node23.emperor: slots=1 max_slots=0 slots_inuse=0 state=UNKNOWN
> =
> Hello World! from process 0 out of 6 on node21.emperor
> Hello World! from process 2 out of 6 on node22.emperor
> Hello World! from process 1 out of 6 on node21.emperor
> Hello World! from process 3 out of 6 on node22.emperor
> Hello World! from process 4 out of 6 on node23.emperor
> Hello World! from process 5 out of 6 on node23.emperor
> 
> These machines are all dual core CPU's.  If a quad core is added to the list
> I get 4 processes on that node. And so on, BUT NOT always.
> 
> Note that the "ALLOCATED NODES" list is NOT obeyed.
> 
> If on the other hand I add "slot=#" to the provided hostfile  it works as 
> expected!
> (the debug output was not included as it is essentially the same as above)
> 
> # awk '{n[$0]++} END {for(i in n)print i,"slots="n[i]}' hostfile.txt > 
> hostfile_slots.txt
> # cat hostfile_slots.txt
> node23.emperor slots=1
> node22.emperor slots=2
> node21.emperor slots=1
> 
> # mpirun --hostfile hostfile_slots.txt mpi_hello
> Hello World! from process 0 out of 4 on node23.emperor
> Hello World! from process 1 out of 4 on node22.emperor
> Hello World! from process 3 out of 4 on node21.emperor
> Hello World! from process 2 out of 4 on node22.emperor
> 
> Or if I convert the hostfile into a comma separated host list it also works.
> 
> # tr '\n' ,  node21.emperor,node22.emperor,node22.emperor,node23.emperor,
> # mpirun --host $(tr '\n' ,  Hello World! from process 0 out of 4 on node21.emperor
> Hello World! from process 1 out of 4 on node22.emperor
> Hello World! from process 3 out of 4 on node23.emperor
> Hello World! from process 2 out of 4 on node22.emperor
> 
> 
> Any help as to why --hostfile does not work as expected and debugged says it
> should be working would be appreciated.
> 
> As you can see I have been studing this problem a long time.  Google has not
> been very helpful.  All I seem to get are man pages, and general help guides.
> 
> 
>   Anthony Thyssen ( System Programmer )
>  --
>   All the books of Power 

Re: [OMPI users] Fwd: Make All error regarding either "Conflicting" or "Previous Declaration" among others

[r...@open-mpi.org]

Err...you might want to ask the MPICH folks. This is the Open MPI mailing list 
:-)

> On Sep 19, 2017, at 7:38 AM, Aragorn Inocencio  
> wrote:
> 
> Good evening,
> 
> Thank you for taking the time to develop and assist in the use of this tool.
> 
> I am trying to install the latest mpich-3.2 version to run Reef3D and my 
> current setup is as follows (sorry I don't know which data is relevant so I'm 
> including as much as possible):
> 
> Windows 7 Professional SP 1
> Installed cygwin using the instructions from the Reef3D manual
> C compiler previously installed on my laptop is Dev C++ 5.8.2, with the 
> Compiler Options saying "Compiler set to configure" "TDM-GCC 4.8.1 64-bit 
> Release"
> echo $shell throws me "/bin/bash"
> So I have successfully done the configure step (step d) in the readme, but 
> running the make command gives me various errors such as: 
> error: redefinition of ‘struct hostent’
> arning: #warning "fd_set and associated macros have been defined in 
> sys/types.  This can cause runtime problems with W32 sockets" [-Wcpp]
> /usr/include/w32api/winsock2.h:976:34: error: conflicting types for ‘connect’
> etc. The main logs are attached for reference as instructed in the readme; 
> however I could not find 
> mpich-3.2/src/pm/hydra/tools/topo/hwloc/hwloc/config.log in the designated 
> folder.  The mv.txt is the output for when running the make V=1 command, so I 
> didn't overwrite the previous m.txt
> 
> Much appreciated.
> -- 
> Ismael Aragorn D. Inocencio
> Civil Engineer
> 
> 
> 
> -- 
> Ismael Aragorn D. Inocencio
> Civil Engineer
>  logs.zip>___
> users mailing list
> users@lists.open-mpi.org
> https://lists.open-mpi.org/mailman/listinfo/users

___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users

Re: [OMPI users] Honor host_aliases file for tight SGE integration

[r...@open-mpi.org]

Hi Reuti

As far as I am concerned, you SGE users “own” the SGE support - so feel free to 
submit a patch!

Ralph

> On Sep 13, 2017, at 9:10 AM, Reuti  wrote:
> 
> Hi,
> 
> I wonder whether it came ever to the discussion, that SGE can have a similar 
> behavior like Torque/PBS regarding the mangling of hostnames. It's similiar 
> to https://github.com/open-mpi/ompi/issues/2328, in the behavior that a node 
> can have multiple network interfaces and each has an unique name. SGE's 
> operation can be routed to a specific network interface by the use of a file:
> 
> $SGE_ROOT/$SGE_CELL/common/host_aliases
> 
> which has the format:
> 
>   
> 
> Hence in the generated $PE_HOSTFILE the name known to SGE is listed, although 
> the `hostname` command provides the real name. Open MPI would in this case 
> start a `qrsh -inherit …` call instead of forking, as it thinks that these 
> are different machines (assuming an allocation_rule of $PE_SLOTS so that the 
> `mpiexec` is supposed to be on the same machine as the started tasks).
> 
> I tried to go the "old" way to provide a start_proc_args to the PE to create 
> a symbolic link to `hostname` in $TMPDIR, so that inside the job script an 
> adjusted `hostname` call is available, but obviously Open MPI calls 
> gethostname() directly and not by an external binary.
> 
> So I mangled the hostname in the created machinefile in the jobscript to feed 
> an "adjusted" $PE_HOSTFILE to Open MPI and then it's working as intended: 
> Open MPI creates forks.
> 
> Does anyone else need such a patch in Open MPI and is it suitable to be 
> included?
> 
> -- Reuti
> 
> PS: Only the headnodes have more than one network interface in our case and 
> hence it's didn't come to my attention up to now, as now there was a need to 
> use also some cores on the headnodes. They are known internally to SGE as 
> "login" and "master", but the external names may be "foo" and "baz" which 
> gethostname() returns.
> ___
> users mailing list
> users@lists.open-mpi.org
> https://lists.open-mpi.org/mailman/listinfo/users

___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users

Re: [OMPI users] OpenMPI 1.10.5 oversubscribing cores

[r...@open-mpi.org]

What you probably want to do is add --cpu-list a,b,c... to each mpirun command, 
where each one lists the cores you want to assign to that job.


> On Sep 8, 2017, at 6:46 AM, twu...@goodyear.com wrote:
> 
> 
> I posted this question last year and we ended up not upgrading to the newer
> openmpi.  Now I need to change to openmpi 1.10.5 and have the same issue.
> 
> Specifically, using 1.4.2, I can run two 12 core jobs on a 24 core node and 
> the
> processes would bind to cores and only have 1 process per core.  ie not
> oversubscribe.
> 
> What I used with 1.4.2 was:
> mpirun --mca mpi_paffinity_alone 1 --mca btl openib,tcp,sm,self ...
> 
> Now with 1.10.5, I have tried multiple combinations of map-to core, bind-to 
> core
> etc and cannot run 2 jobs on the same node without oversubcribing.
> 
> Is there a solution to this?
> 
> Thanks for any info
> tom
> ___
> users mailing list
> users@lists.open-mpi.org
> https://lists.open-mpi.org/mailman/listinfo/users

___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users

Re: [OMPI users] Slot count parameter in hostfile ignored

[r...@open-mpi.org]

It isn’t an issue as there is nothing wrong with OMPI. Your method of joining 
the allocation is a problem. What you have done is to create a job step that 
has only 1 slot/node. We have no choice but to honor that constraint and run 
within it.

What you should be doing is to use salloc to create the allocation. This places 
you inside the main allocation so we can use all of it.


> On Sep 8, 2017, at 1:27 AM, Gilles Gouaillardet <gil...@rist.or.jp> wrote:
> 
> Thanks, now i can reproduce the issue
> 
> 
> Cheers,
> 
> 
> Gilles
> 
> 
> On 9/8/2017 5:20 PM, Maksym Planeta wrote:
>> I run start an interactive allocation and I just noticed that the problem 
>> happens, when I join this allocation from another shell.
>> 
>> Here is how I join:
>> 
>> srun --pty --x11 --jobid=$(squeue -u $USER -o %A | tail -n 1) bash
>> 
>> And here is how I create the allocation:
>> 
>> srun --pty --nodes 8 --ntasks-per-node 24 --mem 50G --time=3:00:00 
>> --partition=haswell --x11 bash
>> 
>> 
>> On 09/08/2017 09:58 AM, Gilles Gouaillardet wrote:
>>> Maxsym,
>>> 
>>> 
>>> can you please post your sbatch script ?
>>> 
>>> fwiw, i am unable to reproduce the issue with the latest v2.x from github.
>>> 
>>> 
>>> by any chance, would you be able to test the latest openmpi 2.1.2rc3 ?
>>> 
>>> 
>>> Cheers,
>>> 
>>> 
>>> Gilles
>>> 
>>> 
>>> On 9/8/2017 4:19 PM, Maksym Planeta wrote:
>>>> Indeed mpirun shows slots=1 per node, but I create allocation with
>>>> --ntasks-per-node 24, so I do have all cores of the node allocated.
>>>> 
>>>> When I use srun I can get all the cores.
>>>> 
>>>> On 09/07/2017 02:12 PM, r...@open-mpi.org wrote:
>>>>> My best guess is that SLURM has only allocated 2 slots, and we
>>>>> respect the RM regardless of what you say in the hostfile. You can
>>>>> check this by adding --display-allocation to your cmd line. You
>>>>> probably need to tell slurm to allocate more cpus/node.
>>>>> 
>>>>> 
>>>>>> On Sep 7, 2017, at 3:33 AM, Maksym Planeta
>>>>>> <mplan...@os.inf.tu-dresden.de> wrote:
>>>>>> 
>>>>>> Hello,
>>>>>> 
>>>>>> I'm trying to tell OpenMPI how many processes per node I want to
>>>>>> use, but mpirun seems to ignore the configuration I provide.
>>>>>> 
>>>>>> I create following hostfile:
>>>>>> 
>>>>>> $ cat hostfile.16
>>>>>> taurusi6344 slots=16
>>>>>> taurusi6348 slots=16
>>>>>> 
>>>>>> And then start the app as follows:
>>>>>> 
>>>>>> $ mpirun --display-map   -machinefile hostfile.16 -np 2 hostname
>>>>>> Data for JOB [42099,1] offset 0
>>>>>> 
>>>>>>    JOB MAP   
>>>>>> 
>>>>>> Data for node: taurusi6344 Num slots: 1Max slots: 0Num
>>>>>> procs: 1
>>>>>>  Process OMPI jobid: [42099,1] App: 0 Process rank: 0 Bound:
>>>>>> socket 0[core 0[hwt 0]], socket 0[core 1[hwt 0]], socket 0[core
>>>>>> 2[hwt 0]], socket 0[core 3[hwt 0]], socket 0[core 4[hwt 0]], socket
>>>>>> 0[core 5[hwt 0]], socket 0[core 6[hwt 0]], socket 0[core 7[hwt 0]],
>>>>>> socket 0[core 8[hwt 0]], socket 0[core 9[hwt 0]], socket 0[core
>>>>>> 10[hwt 0]], socket 0[core 11[hwt
>>>>>> 0]]:[B/B/B/B/B/B/B/B/B/B/B/B][./././././././././././.]
>>>>>> 
>>>>>> Data for node: taurusi6348 Num slots: 1Max slots: 0Num
>>>>>> procs: 1
>>>>>>  Process OMPI jobid: [42099,1] App: 0 Process rank: 1 Bound:
>>>>>> socket 0[core 0[hwt 0]], socket 0[core 1[hwt 0]], socket 0[core
>>>>>> 2[hwt 0]], socket 0[core 3[hwt 0]], socket 0[core 4[hwt 0]], socket
>>>>>> 0[core 5[hwt 0]], socket 0[core 6[hwt 0]], socket 0[core 7[hwt 0]],
>>>>>> socket 0[core 8[hwt 0]], socket 0[core 9[hwt 0]], socket 0[core
>>>>>> 10[hwt 0]], socket 0[core 11[hwt
>>>>>> 0]]:[B/B/B/B/B/B/B/B/B/B/B/B][./././././././././././.]
>>>>>> 
>>>>>> =

Re: [OMPI users] Slot count parameter in hostfile ignored

[r...@open-mpi.org]

My best guess is that SLURM has only allocated 2 slots, and we respect the RM 
regardless of what you say in the hostfile. You can check this by adding 
--display-allocation to your cmd line. You probably need to tell slurm to 
allocate more cpus/node.


> On Sep 7, 2017, at 3:33 AM, Maksym Planeta  
> wrote:
> 
> Hello,
> 
> I'm trying to tell OpenMPI how many processes per node I want to use, but 
> mpirun seems to ignore the configuration I provide.
> 
> I create following hostfile:
> 
> $ cat hostfile.16
> taurusi6344 slots=16
> taurusi6348 slots=16
> 
> And then start the app as follows:
> 
> $ mpirun --display-map   -machinefile hostfile.16 -np 2 hostname
> Data for JOB [42099,1] offset 0
> 
>    JOB MAP   
> 
> Data for node: taurusi6344 Num slots: 1Max slots: 0Num procs: 1
>Process OMPI jobid: [42099,1] App: 0 Process rank: 0 Bound: socket 
> 0[core 0[hwt 0]], socket 0[core 1[hwt 0]], socket 0[core 2[hwt 0]], socket 
> 0[core 3[hwt 0]], socket 0[core 4[hwt 0]], socket 0[core 5[hwt 0]], socket 
> 0[core 6[hwt 0]], socket 0[core 7[hwt 0]], socket 0[core 8[hwt 0]], socket 
> 0[core 9[hwt 0]], socket 0[core 10[hwt 0]], socket 0[core 11[hwt 
> 0]]:[B/B/B/B/B/B/B/B/B/B/B/B][./././././././././././.]
> 
> Data for node: taurusi6348 Num slots: 1Max slots: 0Num procs: 1
>Process OMPI jobid: [42099,1] App: 0 Process rank: 1 Bound: socket 
> 0[core 0[hwt 0]], socket 0[core 1[hwt 0]], socket 0[core 2[hwt 0]], socket 
> 0[core 3[hwt 0]], socket 0[core 4[hwt 0]], socket 0[core 5[hwt 0]], socket 
> 0[core 6[hwt 0]], socket 0[core 7[hwt 0]], socket 0[core 8[hwt 0]], socket 
> 0[core 9[hwt 0]], socket 0[core 10[hwt 0]], socket 0[core 11[hwt 
> 0]]:[B/B/B/B/B/B/B/B/B/B/B/B][./././././././././././.]
> 
> =
> taurusi6344
> taurusi6348
> 
> If I put anything more than 2 in "-np 2", I get following error message:
> 
> $ mpirun --display-map   -machinefile hostfile.16 -np 4 hostname
> --
> There are not enough slots available in the system to satisfy the 4 slots
> that were requested by the application:
>  hostname
> 
> Either request fewer slots for your application, or make more slots available
> for use.
> --
> 
> The OpenMPI version is "mpirun (Open MPI) 2.1.0"
> 
> Also there is SLURM installed with version "slurm 
> 16.05.7-Bull.1.1-20170512-1252"
> 
> Could you help me to enforce OpenMPI to respect slots paremeter?
> -- 
> Regards,
> Maksym Planeta
> 
> ___
> users mailing list
> users@lists.open-mpi.org
> https://lists.open-mpi.org/mailman/listinfo/users

___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users

Re: [OMPI users] Setting LD_LIBRARY_PATH for orted

[r...@open-mpi.org]

I’m afraid not - that only applies the variable to the application, not the 
daemons.

Truly, your only real option is to put something in your .bashrc since you 
cannot modify the configure.

Or, if you are running in a managed environment, you can ask to have your 
resource manager forward your environment to the allocated nodes.

> On Aug 22, 2017, at 9:10 AM, Bennet Fauber  wrote:
> 
> Would
> 
>$ mpirun -x LD_LIBRARY_PATH ...
> 
> work here?  I think from the man page for mpirun that should request
> that it would would export the currently set value of LD_LIBRARY_PATH
> to the remote nodes prior to executing the command there.
> 
> -- bennet
> 
> 
> 
> On Tue, Aug 22, 2017 at 11:55 AM, Jackson, Gary L.
>  wrote:
>> I’m using a build of OpenMPI provided by a third party.
>> 
>> --
>> Gary Jackson, Ph.D.
>> Johns Hopkins University Applied Physics Laboratory
>> 
>> On 8/21/17, 8:04 PM, "users on behalf of Gilles Gouaillardet" 
>>  wrote:
>> 
>>Gary,
>> 
>> 
>>one option (as mentioned in the error message) is to configure Open MPI
>>with --enable-orterun-prefix-by-default.
>> 
>>this will force the build process to use rpath, so you do not have to
>>set LD_LIBRARY_PATH
>> 
>>this is the easiest option, but cannot be used if you plan to relocate
>>the Open MPI installation directory.
>> 
>> 
>>an other option is to use a wrapper for orted.
>> 
>>mpirun --mca orte_launch_agent /.../myorted ...
>> 
>>where myorted is a script that looks like
>> 
>>#!/bin/sh
>> 
>>export LD_LIBRARY_PATH=...
>> 
>>exec /.../bin/orted "$@"
>> 
>> 
>>you can make this setting system-wide by adding the following line to
>>/.../etc/openmpi-mca-params.conf
>> 
>>orte_launch_agent = /.../myorted
>> 
>> 
>>Cheers,
>> 
>> 
>>Gilles
>> 
>> 
>>On 8/22/2017 1:06 AM, Jackson, Gary L. wrote:
>>> 
>>> I’m using a binary distribution of OpenMPI 1.10.2. As linked, it
>>> requires certain shared libraries outside of OpenMPI for orted itself
>>> to start. So, passing in LD_LIBRARY_PATH with the “-x” flag to mpirun
>>> doesn’t do anything:
>>> 
>>> $ mpirun –hostfile ${HOSTFILE} -N 1 -n 2 -x LD_LIBRARY_PATH hostname
>>> 
>>> /path/to/orted: error while loading shared libraries: LIBRARY.so:
>>> cannot open shared object file: No such file or directory
>>> 
>>> --
>>> 
>>> ORTE was unable to reliably start one or more daemons.
>>> 
>>> This usually is caused by:
>>> 
>>> * not finding the required libraries and/or binaries on
>>> 
>>> one or more nodes. Please check your PATH and LD_LIBRARY_PATH
>>> 
>>> settings, or configure OMPI with --enable-orterun-prefix-by-default
>>> 
>>> * lack of authority to execute on one or more specified nodes.
>>> 
>>> Please verify your allocation and authorities.
>>> 
>>> * the inability to write startup files into /tmp
>>> (--tmpdir/orte_tmpdir_base).
>>> 
>>> Please check with your sys admin to determine the correct location to use.
>>> 
>>> * compilation of the orted with dynamic libraries when static are required
>>> 
>>> (e.g., on Cray). Please check your configure cmd line and consider using
>>> 
>>> one of the contrib/platform definitions for your system type.
>>> 
>>> * an inability to create a connection back to mpirun due to a
>>> 
>>> lack of common network interfaces and/or no route found between
>>> 
>>> them. Please check network connectivity (including firewalls
>>> 
>>> and network routing requirements).
>>> 
>>> --
>>> 
>>> How do I get around this cleanly? This works just fine when I set
>>> LD_LIBRARY_PATH in my .bashrc, but I’d rather not pollute that if I
>>> can avoid it.
>>> 
>>> --
>>> 
>>> Gary Jackson, Ph.D.
>>> 
>>> Johns Hopkins University Applied Physics Laboratory
>>> 
>>> 
>>> 
>>> ___
>>> users mailing list
>>> users@lists.open-mpi.org
>>> https://lists.open-mpi.org/mailman/listinfo/users
>> 
>>___
>>users mailing list
>>users@lists.open-mpi.org
>>https://lists.open-mpi.org/mailman/listinfo/users
>> 
>> ___
>> users mailing list
>> users@lists.open-mpi.org
>> https://lists.open-mpi.org/mailman/listinfo/users
> ___
> users mailing list
> users@lists.open-mpi.org
> https://lists.open-mpi.org/mailman/listinfo/users

___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users

Re: [OMPI users] mpirun 2.1.1 refuses to start a Torque 6.1.1.1 job if I change the scheduler to Maui 3.3.1

[r...@open-mpi.org]

sounds to me like your maui scheduler didn’t provide any allocated slots on the 
nodes - did you check $PBS_NODEFILE?

> On Aug 9, 2017, at 12:41 PM, A M  wrote:
> 
> 
> Hello,
> 
> I have just ran into a strange issue with "mpirun". Here is what happened:
> 
> I successfully installed Torque 6.1.1.1 with the plain pbs_sched on a minimal 
> set of 2 IB nodes. Then I added openmpi 2.1.1 compiled with verbs and tm, and 
> have verified that mpirun works as it should with a small "pingpong" program. 
> 
> Here is my Torque minimal jobscript which I used to check the IB message 
> passing:
> 
> #!/bin/sh
> #PBS -o Out
> #PBS -e Err
> #PBS -l nodes=2:ppn=1
> cd $PBS_O_WORKDIR
> mpirun -np 2 -pernode ./pingpong 400
> 
> The job correctly used IB as the default message passing iface and resulted 
> in 3.6 Gb/sec "pingpong" bandwidth which is correct in my case, since the two 
> batch nodes have the QDR HCAs.
> 
> I have then stopped "pbs_sched" and started the Maui 3.3.1 scheduler instead. 
> Serial jobs work without any problem, but the same jobscript is now failing 
> with the following  message:
> 
> 
> Your job has requested more processes than the ppr for this topology can 
> support:
> App: /lustre/work/user/testus/pingpong
> Number of procs:  2
> PPR: 1:node
> Please revise the conflict and try again.
> 
> 
> I then have tried to play with  - -nooversubscribe and "--pernode 2" options, 
> but the error persisted. It looks like the freshmost "mpirun" is getting some 
> information from the latest available Maui scheduler. It is enough to go back 
> to "pbs_sched", and everything works like a charm. I used the preexisting 
> "maui.cfg" file which still works well on the oldish Centos 6 with an old 
> 1.8.5 version of openmpi.  
> 
> Thanks ahead for any hint/comment on how to address this. Are there any other 
> mpirun options to try? Should I try to downgrade openmpi to the latest 1.X 
> series?
> 
> Andy.
>  
> 
>  
> 
> 
> 
> 
> 
> 
> 
> 
> 
> mpirun -np 2 -pernode --mca btl ^tcp ./pingpong 400
> 
> 
> 2.   
> 
>  
> ___
> users mailing list
> users@lists.open-mpi.org
> https://lists.open-mpi.org/mailman/listinfo/users

___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users

Re: [OMPI users] error building openmpi-v2.* with SUN C 5.15 on SuSE Linux

[r...@open-mpi.org]

Should be fixed for 2.x here: https://github.com/open-mpi/ompi/pull/4054 



> On Jul 31, 2017, at 5:56 AM, Siegmar Gross 
>  wrote:
> 
> Hi,
> 
> I've been able to install openmpi-v2.0.x-201707270322-239c439 and
> openmpi-v2.x-201707271804-3b1e9fe on my "SUSE Linux Enterprise Server
> 12.2 (x86_64)" with Sun C 5.14 (Oracle Developer Studio 12.5).
> Unfortunately "make" breaks for both versions with the same error,
> if I use the latest Sun C 5.15 (Oracle Developer Studio 12.5) compiler.
> 
> 
> loki openmpi-v2.0.x-201707270322-239c439-Linux.x86_64.64_cc_12.6 140 tail -18 
> log.make.Linux.x86_64.64_cc | head -7
> "/export2/src/openmpi-2.0.4/openmpi-v2.0.x-201707270322-239c439/opal/include/opal/sys/x86_64/atomic.h",
>  line 270: warning: parameter in inline asm statement unused: %2
> "../../../../../../openmpi-v2.0.x-201707270322-239c439/opal/mca/pmix/pmix112/pmix/src/buffer_ops/open_close.c",
>  line 51: redeclaration must have the same or more restrictive linker 
> scoping: opal_pmix_pmix112_pmix_bfrop
> "../../../../../../openmpi-v2.0.x-201707270322-239c439/opal/mca/pmix/pmix112/pmix/src/buffer_ops/open_close.c",
>  line 401: redeclaration must have the same or more restrictive linker 
> scoping: opal_pmix_pmix112_pmix_value_load
> cc: acomp failed for 
> ../../../../../../openmpi-v2.0.x-201707270322-239c439/opal/mca/pmix/pmix112/pmix/src/buffer_ops/open_close.c
> Makefile:1242: recipe for target 'src/buffer_ops/open_close.lo' failed
> make[4]: *** [src/buffer_ops/open_close.lo] Error 1
> make[4]: Leaving directory 
> '/export2/src/openmpi-2.0.4/openmpi-v2.0.x-201707270322-239c439-Linux.x86_64.64_cc_12.6/opal/mca/pmix/pmix112/pmix'
> loki openmpi-v2.0.x-201707270322-239c439-Linux.x86_64.64_cc_12.6 141
> 
> 
> Perhaps somebody can fix the problem. Thank you very much for your help in
> advance.
> 
> 
> Kind regards
> 
> Siegmar
> ___
> users mailing list
> users@lists.open-mpi.org
> https://rfd.newmexicoconsortium.org/mailman/listinfo/users

___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users

Re: [OMPI users] -host vs -hostfile

[r...@open-mpi.org]

?? Doesn't that tell pbs to allocate 1 node with 2 slots on it? I don't see 
where you get 4 

Sent from my iPad

> On Jul 31, 2017, at 10:00 AM, Mahmood Naderan  wrote:
> 
> OK. The next question is how touse it with torque (PBS)? currently we write 
> this directive
> 
> Nodes=1:ppn=2
> 
> which means 4 threads. Then we omit -np and -hostfile in the mpirun command.
> 
>> On 31 Jul 2017 20:24, "Elken, Tom"  wrote:
>> Hi Mahmood,
>> 
>>  
>> 
>> With the -hostfile case, Open MPI is trying to helpfully run things faster 
>> by keeping both processes on one host.  Ways to avoid this…
>> 
>>  
>> 
>> On the mpirun command line add:
>> 
>>  
>> 
>> -pernode  (runs 1 process per node), oe
>> 
>> -npernode 1 ,   but these two has been deprecated in favor of the wonderful 
>> syntax:
>> 
>> --map-by ppr:1:node
>> 
>>  
>> 
>> Or you could change your hostfile to:
>> 
>> cluster slots=1
>> compute-0-0 slots=1
>>  
>> 
>>  
>> 
>> -Tom
>> 
>>  
>> 
>> From: users [mailto:users-boun...@lists.open-mpi.org] On Behalf Of Mahmood 
>> Naderan
>> Sent: Monday, July 31, 2017 6:47 AM
>> To: Open MPI Users 
>> Subject: [OMPI users] -host vs -hostfile
>> 
>>  
>> 
>> Hi,
>> 
>> I have stuck at a problem which I don't remember that on previous versions. 
>> when I run a test program with -host, it works. I mean, the process spans to 
>> the hosts I specified. However, when I specify -hostfile, it doesn't work!!
>> 
>> mahmood@cluster:mpitest$ /share/apps/computer/openmpi-2.0.1/bin/mpirun -host 
>> compute-0-0,cluster -np 2 a.out
>> 
>> * hwloc 1.11.2 has encountered what looks like an error from the operating 
>> system.
>> *
>> * Package (P#1 cpuset 0x) intersects with NUMANode (P#1 cpuset 
>> 0xff00) without inclusion!
>> * Error occurred in topology.c line 1048
>> *
>> * The following FAQ entry in the hwloc documentation may help:
>> *   What should I do when hwloc reports "operating system" warnings?
>> * Otherwise please report this error message to the hwloc user's mailing 
>> list,
>> * along with the output+tarball generated by the hwloc-gather-topology 
>> script.
>> 
>> Hello world from processor cluster.hpc.org, rank 1 out of 2 processors
>> Hello world from processor compute-0-0.local, rank 0 out of 2 processors
>> mahmood@cluster:mpitest$ cat hosts
>> cluster
>> compute-0-0
>>  
>> mahmood@cluster:mpitest$ /share/apps/computer/openmpi-2.0.1/bin/mpirun 
>> -hostfile hosts -np 2 a.out  
>> 
>> * hwloc 1.11.2 has encountered what looks like an error from the operating 
>> system.
>> *
>> * Package (P#1 cpuset 0x) intersects with NUMANode (P#1 cpuset 
>> 0xff00) without inclusion!
>> * Error occurred in topology.c line 1048
>> *
>> * The following FAQ entry in the hwloc documentation may help:
>> *   What should I do when hwloc reports "operating system" warnings?
>> * Otherwise please report this error message to the hwloc user's mailing 
>> list,
>> * along with the output+tarball generated by the hwloc-gather-topology 
>> script.
>> 
>> Hello world from processor cluster.hpc.org, rank 0 out of 2 processors
>> Hello world from processor cluster.hpc.org, rank 1 out of 2 processors
>> 
>> 
>> how can I resolve that?
>> Regards,
>> Mahmood
>> 
>> 
>> 
>> ___
>> users mailing list
>> users@lists.open-mpi.org
>> https://rfd.newmexicoconsortium.org/mailman/listinfo/users
> ___
> users mailing list
> users@lists.open-mpi.org
> https://rfd.newmexicoconsortium.org/mailman/listinfo/users
___
users mailing list
users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

Re: [OMPI users] Questions about integration with resource distribution systems

[r...@open-mpi.org]

Oh no, that's not right. Mpirun launches daemons using qrsh and those daemons 
spawn the app's procs. SGE has no visibility of the app at all

Sent from my iPad

> On Jul 26, 2017, at 7:46 AM, Kulshrestha, Vipul 
> <vipul_kulshres...@mentor.com> wrote:
> 
> Thanks Reuti & RHC for your responses.
> 
> My application does not relies on the actual value of m_mem_free and I used 
> this as an example, in open source SGE environment, we use mem_free resource.
> 
> Now, I understand that SGE will allocate requested resources (based on qsub 
> options) and then launch mpirun, which starts "a.out" on allocated resouces 
> using 'qrsh -inherit', so that SGE can keep track of all the launched 
> processes.
> 
> I assume LSF integration works in a similar way.
> 
> Regards,
> Vipul
> 
> 
> -Original Message-
> From: users [mailto:users-boun...@lists.open-mpi.org] On Behalf Of Reuti
> Sent: Wednesday, July 26, 2017 9:25 AM
> To: Open MPI Users <users@lists.open-mpi.org>
> Subject: Re: [OMPI users] Questions about integration with resource 
> distribution systems
> 
> 
>> Am 26.07.2017 um 15:09 schrieb r...@open-mpi.org:
>> 
>> mpirun doesn’t get access to that requirement, nor does it need to do so. 
>> SGE will use the requirement when determining the nodes to allocate.
> 
> m_mem_free appears to come from Univa GE and is not part of the open source 
> versions. So I can't comment on this for sure, but it seems to set the memory 
> also in cgroups.
> 
> -- Reuti
> 
> 
>> mpirun just uses the nodes that SGE provides.
>> 
>> What your cmd line does is restrict the entire operation on each node 
>> (daemon + 8 procs) to 40GB of memory. OMPI does not support per-process 
>> restrictions other than binding to cpus.
>> 
>> 
>>> On Jul 26, 2017, at 6:03 AM, Kulshrestha, Vipul 
>>> <vipul_kulshres...@mentor.com> wrote:
>>> 
>>> Thanks for a quick response.
>>> 
>>> I will try building OMPI as suggested.
>>> 
>>> On the integration with unsupported distribution systems, we cannot use 
>>> script based approach, because often these machines don’t have ssh 
>>> permission in customer environment. I will explore the path of writing orte 
>>> component. At this stage, I don’t understand the effort for the same.
>>> 
>>> I guess my question 2 was not understood correctly. I used the below 
>>> command as an example for SGE and want to understand the expected behavior 
>>> for such a command. With the below command, I expect to have 8 copies of 
>>> a.out launched with each copy having access to 40GB of memory. Is that 
>>> correct? I am doubtful, because I don’t understand how mpirun gets access 
>>> to information about RAM requirement.
>>> 
>>> qsub –pe orte 8 –b y –V –l m_mem_free=40G –cwd mpirun –np 8 a.out
>>> 
>>> 
>>> Regards,
>>> Vipul
>>> 
>>> 
>>> 
>>> From: users [mailto:users-boun...@lists.open-mpi.org] On Behalf Of 
>>> r...@open-mpi.org
>>> Sent: Tuesday, July 25, 2017 8:16 PM
>>> To: Open MPI Users <users@lists.open-mpi.org>
>>> Subject: Re: [OMPI users] Questions about integration with resource 
>>> distribution systems
>>> 
>>> 
>>> On Jul 25, 2017, at 3:48 PM, Kulshrestha, Vipul 
>>> <vipul_kulshres...@mentor.com> wrote:
>>> 
>>> I have several questions about integration of openmpi with resource queuing 
>>> systems.
>>> 
>>> 1.
>>> I understand that openmpi supports integration with various resource 
>>> distribution systems such as SGE, LSF, torque etc.
>>> 
>>> I need to build an openmpi application that can interact with variety of 
>>> different resource distribution systems, since different customers have 
>>> different systems. Based on my research, it seems that I need to build a 
>>> different openmpi installation to work, e.g. create an installation of 
>>> opempi with grid and create a different installation of openmpi with LSF. 
>>> Is there a way to build a generic installation of openmpi that can be used 
>>> with more than 1 distribution system by using some generic mechanism?
>>> 
>>> Just to be clear: your application doesn’t depend on the environment in any 
>>> way. Only mpirun does - so if you are distributing an _application_, then 
>>> your question is irrelevant. 
>>> 
>>> If you are distributing OMPI itself, and therefore mpirun, then you can 
>>>

Re: [OMPI users] Questions about integration with resource distribution systems

[r...@open-mpi.org]

mpirun doesn’t get access to that requirement, nor does it need to do so. SGE 
will use the requirement when determining the nodes to allocate. mpirun just 
uses the nodes that SGE provides.

What your cmd line does is restrict the entire operation on each node (daemon + 
8 procs) to 40GB of memory. OMPI does not support per-process restrictions 
other than binding to cpus.


> On Jul 26, 2017, at 6:03 AM, Kulshrestha, Vipul 
> <vipul_kulshres...@mentor.com> wrote:
> 
> Thanks for a quick response.
>  
> I will try building OMPI as suggested.
>  
> On the integration with unsupported distribution systems, we cannot use 
> script based approach, because often these machines don’t have ssh permission 
> in customer environment. I will explore the path of writing orte component. 
> At this stage, I don’t understand the effort for the same.
>  
> I guess my question 2 was not understood correctly. I used the below command 
> as an example for SGE and want to understand the expected behavior for such a 
> command. With the below command, I expect to have 8 copies of a.out launched 
> with each copy having access to 40GB of memory. Is that correct? I am 
> doubtful, because I don’t understand how mpirun gets access to information 
> about RAM requirement.
>  
> qsub –pe orte 8 –b y –V –l m_mem_free=40G –cwd mpirun –np 8 a.out
>  
>  
> Regards,
> Vipul
>  
>  
>   <>
> From: users [mailto:users-boun...@lists.open-mpi.org] On Behalf Of 
> r...@open-mpi.org
> Sent: Tuesday, July 25, 2017 8:16 PM
> To: Open MPI Users <users@lists.open-mpi.org>
> Subject: Re: [OMPI users] Questions about integration with resource 
> distribution systems
>  
>  
> On Jul 25, 2017, at 3:48 PM, Kulshrestha, Vipul <vipul_kulshres...@mentor.com 
> <mailto:vipul_kulshres...@mentor.com>> wrote:
>  
> I have several questions about integration of openmpi with resource queuing 
> systems.
>  
> 1.
> I understand that openmpi supports integration with various resource 
> distribution systems such as SGE, LSF, torque etc.
>  
> I need to build an openmpi application that can interact with variety of 
> different resource distribution systems, since different customers have 
> different systems. Based on my research, it seems that I need to build a 
> different openmpi installation to work, e.g. create an installation of opempi 
> with grid and create a different installation of openmpi with LSF. Is there a 
> way to build a generic installation of openmpi that can be used with more 
> than 1 distribution system by using some generic mechanism?
>  
> Just to be clear: your application doesn’t depend on the environment in any 
> way. Only mpirun does - so if you are distributing an _application_, then 
> your question is irrelevant. 
>  
> If you are distributing OMPI itself, and therefore mpirun, then you can build 
> the various components if you first install the headers for that environment 
> on your system. It means that you need one machine where all those resource 
> managers at least have their headers installed on it. Then configure OMPI 
> --with-xxx pointing to each of the RM’s headers so all the components get 
> built. When the binary hits your customer’s machine, only those components 
> that have active libraries present will execute.
>  
>  
> 2.
> For integration with LSF/grid, how would I specify the memory (RAM) 
> requirement (or some other parameter) to bsub/qsub, when launching mpirun 
> command? Will something like below work to ensure that each of the 8 copies 
> of a.out have 40 GB memory reserved for them by grid engine?
>  
> qsub –pe orte 8 –b y –V –l m_mem_free=40G –cwd mpirun –np 8 a.out
>  
> You’ll have to provide something that is environment dependent, I’m afraid - 
> there is no standard out there.
> 
> 
>  
> 3.
> Some of our customers use custom distribution engine (some 
> non-industry-standard distribution engine). How can I integrate my openmpi  
> application with such system? I would think that it should be possible to do 
> that if openmpi launched/managed interaction with the distribution engine 
> using some kind of generic mechanism (say, use a configurable command to 
> launch, monitor, kill a job and then allow specification of a plugin define 
> these operations with commands specific to the distribution engine being in 
> use). Does such integration exist in openmpi?
>  
> Easiest solution is to write a script that reads the allocation and dumps it 
> into a file, and then provide that file as your hostfile on the mpirun cmd 
> line (or in the environment). We will then use ssh to perform the launch. 
> Otherwise, you’ll need to write at least an orte/mca/ras component to

Re: [OMPI users] Questions about integration with resource distribution systems

[r...@open-mpi.org]

> On Jul 25, 2017, at 3:48 PM, Kulshrestha, Vipul 
>  wrote:
> 
> I have several questions about integration of openmpi with resource queuing 
> systems.
>  
> 1.
> I understand that openmpi supports integration with various resource 
> distribution systems such as SGE, LSF, torque etc.
>  
> I need to build an openmpi application that can interact with variety of 
> different resource distribution systems, since different customers have 
> different systems. Based on my research, it seems that I need to build a 
> different openmpi installation to work, e.g. create an installation of opempi 
> with grid and create a different installation of openmpi with LSF. Is there a 
> way to build a generic installation of openmpi that can be used with more 
> than 1 distribution system by using some generic mechanism?

Just to be clear: your application doesn’t depend on the environment in any 
way. Only mpirun does - so if you are distributing an _application_, then your 
question is irrelevant. 

If you are distributing OMPI itself, and therefore mpirun, then you can build 
the various components if you first install the headers for that environment on 
your system. It means that you need one machine where all those resource 
managers at least have their headers installed on it. Then configure OMPI 
--with-xxx pointing to each of the RM’s headers so all the components get 
built. When the binary hits your customer’s machine, only those components that 
have active libraries present will execute.

>  
> 2.
> For integration with LSF/grid, how would I specify the memory (RAM) 
> requirement (or some other parameter) to bsub/qsub, when launching mpirun 
> command? Will something like below work to ensure that each of the 8 copies 
> of a.out have 40 GB memory reserved for them by grid engine?
>  
> qsub –pe orte 8 –b y –V –l m_mem_free=40G –cwd mpirun –np 8 a.out

You’ll have to provide something that is environment dependent, I’m afraid - 
there is no standard out there.

>  
> 3.
> Some of our customers use custom distribution engine (some 
> non-industry-standard distribution engine). How can I integrate my openmpi  
> application with such system? I would think that it should be possible to do 
> that if openmpi launched/managed interaction with the distribution engine 
> using some kind of generic mechanism (say, use a configurable command to 
> launch, monitor, kill a job and then allow specification of a plugin define 
> these operations with commands specific to the distribution engine being in 
> use). Does such integration exist in openmpi?

Easiest solution is to write a script that reads the allocation and dumps it 
into a file, and then provide that file as your hostfile on the mpirun cmd line 
(or in the environment). We will then use ssh to perform the launch. Otherwise, 
you’ll need to write at least an orte/mca/ras component to get the allocation, 
and possibly an orte/mca/plm component if you want to use the native launch 
mechanism in place of ssh.

>  
>  
> Thanks,
> Vipul
>  
>  
> ___
> users mailing list
> users@lists.open-mpi.org 
> https://rfd.newmexicoconsortium.org/mailman/listinfo/users 
> 
___
users mailing list
users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

Re: [OMPI users] Issue handling SIGUSR1 in OpenMPI

[r...@open-mpi.org]

Again, you are sending the signal to just the one process whose pid you 
specified. We don’t pick that signal up and propagate it. If you signal the pid 
of mpiexec itself, then you’d see every proc report it.

> On Jul 25, 2017, at 11:40 AM, Marc Cooper <marccooper2...@gmail.com> wrote:
> 
> Even this method of raising signal from user to mpiexec results in signal 
> handling by only one process. I've modified my earlier example where each 
> process publishes its pid, and I capture the pid and raise the signal using 
> 'kill -SIGUSR1 ' from another terminal. 
> 
> // test.c
> 
> void handle_signal(int signal){ 
> if(SIGNAL==SIGUSR1)
> printf("received SIGUSR1 signal \n");
> }
> int main(){
> MPI_Init(NULL, NULL);
> 
>int my_rank;
>MPI_Comm_rank(MPI_COMM_WORLD, _rank);
> 
>signal(SIGUSR1, handle_signal);
> 
>printf("My pid is: %d\n", getpid());
> 
> for (;;) {
> printf("\nSleeping for 10 seconds\n");
> sleep(10); 
> 
> MPI_Finalize();
> }
> 
> When I run with 3 processes using mpirun -np 3 ./test, I expect the statement 
> 'received SIGUSR1 signal' twice, but it prints just once. What am I missing 
> here?
>  
> 
> 
> On 25 July 2017 at 11:17, r...@open-mpi.org <mailto:r...@open-mpi.org> 
> <r...@open-mpi.org <mailto:r...@open-mpi.org>> wrote:
> I’m afraid we don’t currently support that use-case. We forward signals sent 
> by the user to mpiexec (i.e., the user “hits” mpiexec with a signal), but we 
> don’t do anything to support an application proc attempting to raise a signal 
> and asking it to be propagated.
> 
> If you are using OMPI master, or the soon-to-be-released v3.0, then you might 
> be able to do what you seek using the PMIx event notification system.
> 
>> On Jul 25, 2017, at 10:15 AM, Marc Cooper <marccooper2...@gmail.com 
>> <mailto:marccooper2...@gmail.com>> wrote:
>> 
>> Hi all,
>> 
>> I'm working to understand signal handling in OpenMPI. I read that "Open MPI 
>> will forward SIGUSR1 and SIGUSR2 from mpiexec to the other processes". My 
>> question is that is this feature enabled by default installation.
>> 
>> The scenario is that one MPI process raises a SIGUSR1, which has to be 
>> detected by 'orted' which is then forwarded to other processes.
>> 
>> In my test code, I define a custom signal handler for SIGUSR1 and register 
>> this signal handler accordingly. I send a signal by using kill() or raise(). 
>> I assume that ORTE daemon will receive this signal and has to forward this 
>> signal to the remaining processes.
>> 
>> // test.c
>> 
>> void handle_signal(int signal){ 
>> if(SIGNAL==SIGUSR1)
>> printf("received SIGUSR1 signal \n");
>> }
>> int main(){
>> MPI_Init(NULL, NULL);
>> 
>>int my_rank;
>>MPI_Comm_rank(MPI_COMM_WORLD, _rank);
>> 
>> signal(SIGUSR1, handle_signal);
>> 
>> if(my_rank == 1) // process with rank 1 raises SIGUSR1
>>  kill(getpid(), SIGUSR1);
>> 
>> MPI_Finalize();
>> }
>> 
>> If I run this as
>> mpirun -np 3 ./test
>> 
>> I would expect to have the statement printed twice from the other two 
>> processes. But when I run this code, it only prints once, and that too from 
>> ORTE HNP, unlike the application processes. Do I need to call any other API 
>> on orted explicitly pass this signal, so that the application processes 
>> receive the SIGUSR1.
>> 
>> -
>> Marc
>> ___
>> users mailing list
>> users@lists.open-mpi.org <mailto:users@lists.open-mpi.org>
>> https://rfd.newmexicoconsortium.org/mailman/listinfo/users 
>> <https://rfd.newmexicoconsortium.org/mailman/listinfo/users>
> 
> ___
> users mailing list
> users@lists.open-mpi.org <mailto:users@lists.open-mpi.org>
> https://rfd.newmexicoconsortium.org/mailman/listinfo/users 
> <https://rfd.newmexicoconsortium.org/mailman/listinfo/users>
> 
> ___
> users mailing list
> users@lists.open-mpi.org
> https://rfd.newmexicoconsortium.org/mailman/listinfo/users

___
users mailing list
users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

Re: [OMPI users] Issue handling SIGUSR1 in OpenMPI

[r...@open-mpi.org]

I’m afraid we don’t currently support that use-case. We forward signals sent by 
the user to mpiexec (i.e., the user “hits” mpiexec with a signal), but we don’t 
do anything to support an application proc attempting to raise a signal and 
asking it to be propagated.

If you are using OMPI master, or the soon-to-be-released v3.0, then you might 
be able to do what you seek using the PMIx event notification system.

> On Jul 25, 2017, at 10:15 AM, Marc Cooper  wrote:
> 
> Hi all,
> 
> I'm working to understand signal handling in OpenMPI. I read that "Open MPI 
> will forward SIGUSR1 and SIGUSR2 from mpiexec to the other processes". My 
> question is that is this feature enabled by default installation.
> 
> The scenario is that one MPI process raises a SIGUSR1, which has to be 
> detected by 'orted' which is then forwarded to other processes.
> 
> In my test code, I define a custom signal handler for SIGUSR1 and register 
> this signal handler accordingly. I send a signal by using kill() or raise(). 
> I assume that ORTE daemon will receive this signal and has to forward this 
> signal to the remaining processes.
> 
> // test.c
> 
> void handle_signal(int signal){ 
> if(SIGNAL==SIGUSR1)
> printf("received SIGUSR1 signal \n");
> }
> int main(){
> MPI_Init(NULL, NULL);
> 
>int my_rank;
>MPI_Comm_rank(MPI_COMM_WORLD, _rank);
> 
> signal(SIGUSR1, handle_signal);
> 
> if(my_rank == 1) // process with rank 1 raises SIGUSR1
>  kill(getpid(), SIGUSR1);
> 
> MPI_Finalize();
> }
> 
> If I run this as
> mpirun -np 3 ./test
> 
> I would expect to have the statement printed twice from the other two 
> processes. But when I run this code, it only prints once, and that too from 
> ORTE HNP, unlike the application processes. Do I need to call any other API 
> on orted explicitly pass this signal, so that the application processes 
> receive the SIGUSR1.
> 
> -
> Marc
> ___
> users mailing list
> users@lists.open-mpi.org
> https://rfd.newmexicoconsortium.org/mailman/listinfo/users

___
users mailing list
users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

Re: [OMPI users] OpenMPI 2.1.1, --map-to socket, application context files

[r...@open-mpi.org]

Well, yes and no. Yes, your cpu loads will balance better across nodes 
(balancing across sockets doesn’t do much for you). However, your overall 
application performance may be the poorest in that arrangement if your app uses 
a lot of communication as the layout minimizes the use of shared memory.

Laying out an app requires a little thought about its characteristics. If it is 
mostly compute with a little communication, then spreading the procs out makes 
the most sense. If it has a lot of communication, then compressing the procs 
into the minimum space makes the most sense. This is the most commonly used 
layout.

I haven’t looked at app context files in ages, but I think you could try this:

-np 1 afftest01.exe; -np 1 afftest01.exe


> On Jun 30, 2017, at 5:03 AM, Ted Sussman <ted.suss...@adina.com> wrote:
> 
> Hello Ralph,
> 
> Thank you for your comments.
> 
> My understanding, from reading Jeff's blog on V1.5 processor affinity, is 
> that the bindings in 
> Example 1 balance the load better than the bindings in Example 2.
> 
> Therefore I would like to obtain the bindings in Example 1, but using Open 
> MPI 2.1.1, and 
> using application context files.
> 
> How can I do this?
> 
> Sincerely,
> 
> Ted Sussman
> 
> On 29 Jun 2017 at 19:09, r...@open-mpi.org wrote:
> 
>> 
>> It´s a difficult call to make as to which is the correct behavior. In 
>> Example 1, you are executing a 
>> single app_context that has two procs in it. In Example 2, you are executing 
>> two app_contexts, 
>> each with a single proc in it.
>> 
>> Now some people say that the two should be treated the same, with the second 
>> app_context in 
>> Example 2 being mapped starting from the end of the first app_context. In 
>> this model, a 
>> comm_spawn would also start from the end of the earlier app_context, and 
>> thus the new proc 
>> would not be on the same node (or socket, in this case) as its parent.
>> 
>> Other people argue for the opposite behavior - that each app_context should 
>> start from the first 
>> available slot in the allocation. In that model, a comm_spawn would result 
>> in the first child 
>> occupying the same node (or socket) as its parent, assuming an available 
>> slot.
>> 
>> We´ve bounced around a bit on the behavior over the years as different 
>> groups voiced their 
>> opinions. OMPI 1.4.3 is _very_ old and fell in the prior camp, while 2.1.1 
>> is just released and is in 
>> the second camp. I honestly don´t recall where the change occurred, or even 
>> how consistent we 
>> have necessarily been over the years. It isn´t something that people raise 
>> very often.
>> 
>> I´ve pretty much resolved to leave the default behavior as it currently 
>> sits, but plan to add an option 
>> to support the alternative behavior as there seems no clear cut consensus in 
>> the user community 
>> for this behavior. Not sure when I´ll get to it - definitely not for the 2.x 
>> series, and maybe not for 3.x 
>> since that is about to be released.
>> 
>>On Jun 29, 2017, at 11:24 AM, Ted Sussman <ted.suss...@adina.com> wrote:
>> 
>>Hello all,
>> 
>>Today I have a problem with the --map-to socket feature of Open MPI 2.1.1 
>> when used with 
>>application context files.
>> 
>>In the examples below, I am testing on a 2 socket computer, each socket 
>> with 4 cores.
>> 
>>---
>> 
>>Example 1:
>> 
>>.../openmpi-2.1.1/bin/mpirun --report-bindings \
>>-map-by socket \
>>-np 2 \
>>afftest01.exe
>> 
>>returns
>> 
>>...MCW rank 0 bound to socket 0 ... : [B/B/B/B][./././.]
>>...MCW rank 1 bound to socket 1 ... : [./././.][B/B/B/B]
>> 
>>which is what I would expect.
>> 
>>---
>> 
>>Example 2:
>> 
>>Create appfile as:
>> 
>>-np 1 afftest01.exe
>>-np 1 afftest01.exe
>> 
>>Then
>> 
>>.../openmpi-2.1.1/bin/mpirun --report-bindings \
>>-map-by socket \
>>-app appfile
>> 
>>returns
>> 
>>...MCW rank 0 bound to socket 0 ... : [B/B/B/B][./././.]
>>...MCW rank 1 bound to socket 0 ... : [B/B/B/B][./././.]
>> 
>>which is not what I expect. I expect the same bindings as in Example 1.
>> 
>>---
>> 
>>Example 3:
>> 
>>Using the same appfile as in Example 2,
>> 
>>.../openmpi-1.4.3/bin/mpirun --report-bind

Re: [OMPI users] OpenMPI 2.1.1, --map-to socket, application context files

[r...@open-mpi.org]

It’s a difficult call to make as to which is the correct behavior. In Example 
1, you are executing a single app_context that has two procs in it. In Example 
2, you are executing two app_contexts, each with a single proc in it.

Now some people say that the two should be treated the same, with the second 
app_context in Example 2 being mapped starting from the end of the first 
app_context. In this model, a comm_spawn would also start from the end of the 
earlier app_context, and thus the new proc would not be on the same node (or 
socket, in this case) as its parent.

Other people argue for the opposite behavior - that each app_context should 
start from the first available slot in the allocation. In that model, a 
comm_spawn would result in the first child occupying the same node (or socket) 
as its parent, assuming an available slot.

We’ve bounced around a bit on the behavior over the years as different groups 
voiced their opinions. OMPI 1.4.3 is _very_ old and fell in the prior camp, 
while 2.1.1 is just released and is in the second camp. I honestly don’t recall 
where the change occurred, or even how consistent we have necessarily been over 
the years. It isn’t something that people raise very often.

I’ve pretty much resolved to leave the default behavior as it currently sits, 
but plan to add an option to support the alternative behavior as there seems no 
clear cut consensus in the user community for this behavior. Not sure when I’ll 
get to it - definitely not for the 2.x series, and maybe not for 3.x since that 
is about to be released.

> On Jun 29, 2017, at 11:24 AM, Ted Sussman  wrote:
> 
> Hello all,
> 
> Today I have a problem with the --map-to socket feature of Open MPI 2.1.1 
> when used with application context files.
> 
> In the examples below, I am testing on a 2 socket computer, each socket with 
> 4 cores.
> 
> ---
> 
> Example 1:
> 
> .../openmpi-2.1.1/bin/mpirun --report-bindings \
> -map-by socket \
> -np 2 \
> afftest01.exe
> 
> returns
> 
> ...MCW rank 0 bound to socket 0 ... : [B/B/B/B][./././.]
> ...MCW rank 1 bound to socket 1 ... : [./././.][B/B/B/B]
> 
> which is what I would expect.
> 
> ---
> 
> Example 2:
> 
> Create appfile as:
> 
> -np 1 afftest01.exe
> -np 1 afftest01.exe
> 
> Then
> 
> .../openmpi-2.1.1/bin/mpirun --report-bindings \
> -map-by socket \
> -app appfile
> 
> returns
> 
> ...MCW rank 0 bound to socket 0 ... : [B/B/B/B][./././.]
> ...MCW rank 1 bound to socket 0 ... : [B/B/B/B][./././.]
> 
> which is not what I expect. I expect the same bindings as in Example 1.
> 
> ---
> 
> Example 3:
> 
> Using the same appfile as in Example 2,
> 
> .../openmpi-1.4.3/bin/mpirun --report-bindings \
> -bysocket --bind-to-core  \
> -app appfile
> 
> returns
> 
> ... odls:default:fork binding child ... to socket 0 cpus 0002
> ... odls:default:fork binding child ... to socket 1 cpus 0001
> 
> which is what I would expect.  Here I use --bind-to-core just to get the 
> bindings printed.
> 
> ---
> 
> The examples show that the --map-by socket feature does not work as expected 
> when application context files are used.  However the older -bysocket feature 
> worked as expected in OpenMPI 1.4.3 when application context files are used.
> 
> If I am using the wrong syntax in Example 2, please let me know.
> 
> Sincerely,
> 
> Ted Sussman
> 
>   
> ___
> users mailing list
> users@lists.open-mpi.org
> https://rfd.newmexicoconsortium.org/mailman/listinfo/users

___
users mailing list
users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

Re: [OMPI users] Node failure handling

[r...@open-mpi.org]

Okay, this should fix it - https://github.com/open-mpi/ompi/pull/3771 
<https://github.com/open-mpi/ompi/pull/3771>

> On Jun 27, 2017, at 6:31 AM, r...@open-mpi.org wrote:
> 
> Actually, the error message is coming from mpirun to indicate that it lost 
> connection to one (or more) of its daemons. This happens because slurm only 
> knows about the remote daemons - mpirun was started outside of “srun”, and so 
> slurm doesn’t know it exists. Thus, when slurm kills the job, it only kills 
> the daemons on the compute nodes, not mpirun. As a result, we always see that 
> error message.
> 
> The capability should exist as an option - it used to, but probably has 
> fallen into disrepair. I’ll see if I can bring it back.
> 
>> On Jun 27, 2017, at 3:35 AM, George Bosilca <bosi...@icl.utk.edu 
>> <mailto:bosi...@icl.utk.edu>> wrote:
>> 
>> I would also be interested in having the slurm keep the remaining processes 
>> around, we have been struggling with this on many of the NERSC machines. 
>> That being said the error message comes from orted, and it suggest that they 
>> are giving up because they lose connection to a peer. I was not aware that 
>> this capability exists in the master version of ORTE, but if it does then it 
>> makes our life easier.
>> 
>>   George.
>> 
>> 
>> On Tue, Jun 27, 2017 at 6:14 AM, r...@open-mpi.org 
>> <mailto:r...@open-mpi.org> <r...@open-mpi.org <mailto:r...@open-mpi.org>> 
>> wrote:
>> Let me poke at it a bit tomorrow - we should be able to avoid the abort. 
>> It’s a bug if we can’t.
>> 
>> > On Jun 26, 2017, at 7:39 PM, Tim Burgess <ozburgess+o...@gmail.com 
>> > <mailto:ozburgess%2bo...@gmail.com>> wrote:
>> >
>> > Hi Ralph,
>> >
>> > Thanks for the quick response.
>> >
>> > Just tried again not under slurm, but the same result... (though I
>> > just did kill -9 orted on the remote node this time)
>> >
>> > Any ideas?  Do you think my multiple-mpirun idea is worth trying?
>> >
>> > Cheers,
>> > Tim
>> >
>> >
>> > ```
>> > [user@bud96 mpi_resilience]$
>> > /d/home/user/2017/openmpi-master-20170608/bin/mpirun --mca plm rsh
>> > --host bud96,pnod0331 -np 2 --npernode 1 --enable-recovery
>> > --debug-daemons $(pwd)/test
>> > ( some output from job here )
>> > ( I then do kill -9 `pgrep orted`  on pnod0331 )
>> > bash: line 1: 161312 Killed
>> > /d/home/user/2017/openmpi-master-20170608/bin/orted -mca
>> > orte_debug_daemons "1" -mca ess "env" -mca ess_base_jobid "581828608"
>> > -mca ess_base_vpid 1 -mca ess_base_num_procs "2" -mca orte_node_regex
>> > "bud[2:96],pnod[4:331]@0(2)" -mca orte_hnp_uri
>> > "581828608.0;tcp://172.16.251.96 
>> > <http://172.16.251.96/>,172.31.1.254:58250 <http://172.31.1.254:58250/>" 
>> > -mca plm "rsh"
>> > -mca rmaps_ppr_n_pernode "1" -mca orte_enable_recovery "1"
>> > --
>> > ORTE has lost communication with a remote daemon.
>> >
>> >  HNP daemon   : [[8878,0],0] on node bud96
>> >  Remote daemon: [[8878,0],1] on node pnod0331
>> >
>> > This is usually due to either a failure of the TCP network
>> > connection to the node, or possibly an internal failure of
>> > the daemon itself. We cannot recover from this failure, and
>> > therefore will terminate the job.
>> > --
>> > [bud96:20652] [[8878,0],0] orted_cmd: received halt_vm cmd
>> > [bud96:20652] [[8878,0],0] orted_cmd: all routes and children gone - 
>> > exiting
>> > ```
>> >
>> > On 27 June 2017 at 12:19, r...@open-mpi.org <mailto:r...@open-mpi.org> 
>> > <r...@open-mpi.org <mailto:r...@open-mpi.org>> wrote:
>> >> Ah - you should have told us you are running under slurm. That does 
>> >> indeed make a difference. When we launch the daemons, we do so with "srun 
>> >> --kill-on-bad-exit” - this means that slurm automatically kills the job 
>> >> if any daemon terminates. We take that measure to avoid leaving zombies 
>> >> behind in the event of a failure.
>> >>
>> >> Try adding “-mca plm rsh” to your mpirun cmd line. This will use the rsh 
>> >> launcher instead of the slurm one, which gives

Re: [OMPI users] Node failure handling

[r...@open-mpi.org]

Actually, the error message is coming from mpirun to indicate that it lost 
connection to one (or more) of its daemons. This happens because slurm only 
knows about the remote daemons - mpirun was started outside of “srun”, and so 
slurm doesn’t know it exists. Thus, when slurm kills the job, it only kills the 
daemons on the compute nodes, not mpirun. As a result, we always see that error 
message.

The capability should exist as an option - it used to, but probably has fallen 
into disrepair. I’ll see if I can bring it back.

> On Jun 27, 2017, at 3:35 AM, George Bosilca <bosi...@icl.utk.edu> wrote:
> 
> I would also be interested in having the slurm keep the remaining processes 
> around, we have been struggling with this on many of the NERSC machines. That 
> being said the error message comes from orted, and it suggest that they are 
> giving up because they lose connection to a peer. I was not aware that this 
> capability exists in the master version of ORTE, but if it does then it makes 
> our life easier.
> 
>   George.
> 
> 
> On Tue, Jun 27, 2017 at 6:14 AM, r...@open-mpi.org <mailto:r...@open-mpi.org> 
> <r...@open-mpi.org <mailto:r...@open-mpi.org>> wrote:
> Let me poke at it a bit tomorrow - we should be able to avoid the abort. It’s 
> a bug if we can’t.
> 
> > On Jun 26, 2017, at 7:39 PM, Tim Burgess <ozburgess+o...@gmail.com 
> > <mailto:ozburgess%2bo...@gmail.com>> wrote:
> >
> > Hi Ralph,
> >
> > Thanks for the quick response.
> >
> > Just tried again not under slurm, but the same result... (though I
> > just did kill -9 orted on the remote node this time)
> >
> > Any ideas?  Do you think my multiple-mpirun idea is worth trying?
> >
> > Cheers,
> > Tim
> >
> >
> > ```
> > [user@bud96 mpi_resilience]$
> > /d/home/user/2017/openmpi-master-20170608/bin/mpirun --mca plm rsh
> > --host bud96,pnod0331 -np 2 --npernode 1 --enable-recovery
> > --debug-daemons $(pwd)/test
> > ( some output from job here )
> > ( I then do kill -9 `pgrep orted`  on pnod0331 )
> > bash: line 1: 161312 Killed
> > /d/home/user/2017/openmpi-master-20170608/bin/orted -mca
> > orte_debug_daemons "1" -mca ess "env" -mca ess_base_jobid "581828608"
> > -mca ess_base_vpid 1 -mca ess_base_num_procs "2" -mca orte_node_regex
> > "bud[2:96],pnod[4:331]@0(2)" -mca orte_hnp_uri
> > "581828608.0;tcp://172.16.251.96 <http://172.16.251.96/>,172.31.1.254:58250 
> > <http://172.31.1.254:58250/>" -mca plm "rsh"
> > -mca rmaps_ppr_n_pernode "1" -mca orte_enable_recovery "1"
> > --
> > ORTE has lost communication with a remote daemon.
> >
> >  HNP daemon   : [[8878,0],0] on node bud96
> >  Remote daemon: [[8878,0],1] on node pnod0331
> >
> > This is usually due to either a failure of the TCP network
> > connection to the node, or possibly an internal failure of
> > the daemon itself. We cannot recover from this failure, and
> > therefore will terminate the job.
> > --
> > [bud96:20652] [[8878,0],0] orted_cmd: received halt_vm cmd
> > [bud96:20652] [[8878,0],0] orted_cmd: all routes and children gone - exiting
> > ```
> >
> > On 27 June 2017 at 12:19, r...@open-mpi.org <mailto:r...@open-mpi.org> 
> > <r...@open-mpi.org <mailto:r...@open-mpi.org>> wrote:
> >> Ah - you should have told us you are running under slurm. That does indeed 
> >> make a difference. When we launch the daemons, we do so with "srun 
> >> --kill-on-bad-exit” - this means that slurm automatically kills the job if 
> >> any daemon terminates. We take that measure to avoid leaving zombies 
> >> behind in the event of a failure.
> >>
> >> Try adding “-mca plm rsh” to your mpirun cmd line. This will use the rsh 
> >> launcher instead of the slurm one, which gives you more control.
> >>
> >>> On Jun 26, 2017, at 6:59 PM, Tim Burgess <ozburgess+o...@gmail.com 
> >>> <mailto:ozburgess%2bo...@gmail.com>> wrote:
> >>>
> >>> Hi Ralph, George,
> >>>
> >>> Thanks very much for getting back to me.  Alas, neither of these
> >>> options seem to accomplish the goal.  Both in OpenMPI v2.1.1 and on a
> >>> recent master (7002535), with slurm's "--no-kill" and openmpi's
> >>> "--enable-recovery", once the node reboots one gets the foll

Re: [OMPI users] Node failure handling

[r...@open-mpi.org]

Let me poke at it a bit tomorrow - we should be able to avoid the abort. It’s a 
bug if we can’t.

> On Jun 26, 2017, at 7:39 PM, Tim Burgess <ozburgess+o...@gmail.com> wrote:
> 
> Hi Ralph,
> 
> Thanks for the quick response.
> 
> Just tried again not under slurm, but the same result... (though I
> just did kill -9 orted on the remote node this time)
> 
> Any ideas?  Do you think my multiple-mpirun idea is worth trying?
> 
> Cheers,
> Tim
> 
> 
> ```
> [user@bud96 mpi_resilience]$
> /d/home/user/2017/openmpi-master-20170608/bin/mpirun --mca plm rsh
> --host bud96,pnod0331 -np 2 --npernode 1 --enable-recovery
> --debug-daemons $(pwd)/test
> ( some output from job here )
> ( I then do kill -9 `pgrep orted`  on pnod0331 )
> bash: line 1: 161312 Killed
> /d/home/user/2017/openmpi-master-20170608/bin/orted -mca
> orte_debug_daemons "1" -mca ess "env" -mca ess_base_jobid "581828608"
> -mca ess_base_vpid 1 -mca ess_base_num_procs "2" -mca orte_node_regex
> "bud[2:96],pnod[4:331]@0(2)" -mca orte_hnp_uri
> "581828608.0;tcp://172.16.251.96,172.31.1.254:58250" -mca plm "rsh"
> -mca rmaps_ppr_n_pernode "1" -mca orte_enable_recovery "1"
> --
> ORTE has lost communication with a remote daemon.
> 
>  HNP daemon   : [[8878,0],0] on node bud96
>  Remote daemon: [[8878,0],1] on node pnod0331
> 
> This is usually due to either a failure of the TCP network
> connection to the node, or possibly an internal failure of
> the daemon itself. We cannot recover from this failure, and
> therefore will terminate the job.
> --------------
> [bud96:20652] [[8878,0],0] orted_cmd: received halt_vm cmd
> [bud96:20652] [[8878,0],0] orted_cmd: all routes and children gone - exiting
> ```
> 
> On 27 June 2017 at 12:19, r...@open-mpi.org <r...@open-mpi.org> wrote:
>> Ah - you should have told us you are running under slurm. That does indeed 
>> make a difference. When we launch the daemons, we do so with "srun 
>> --kill-on-bad-exit” - this means that slurm automatically kills the job if 
>> any daemon terminates. We take that measure to avoid leaving zombies behind 
>> in the event of a failure.
>> 
>> Try adding “-mca plm rsh” to your mpirun cmd line. This will use the rsh 
>> launcher instead of the slurm one, which gives you more control.
>> 
>>> On Jun 26, 2017, at 6:59 PM, Tim Burgess <ozburgess+o...@gmail.com> wrote:
>>> 
>>> Hi Ralph, George,
>>> 
>>> Thanks very much for getting back to me.  Alas, neither of these
>>> options seem to accomplish the goal.  Both in OpenMPI v2.1.1 and on a
>>> recent master (7002535), with slurm's "--no-kill" and openmpi's
>>> "--enable-recovery", once the node reboots one gets the following
>>> error:
>>> 
>>> ```
>>> --
>>> ORTE has lost communication with a remote daemon.
>>> 
>>> HNP daemon   : [[58323,0],0] on node pnod0330
>>> Remote daemon: [[58323,0],1] on node pnod0331
>>> 
>>> This is usually due to either a failure of the TCP network
>>> connection to the node, or possibly an internal failure of
>>> the daemon itself. We cannot recover from this failure, and
>>> therefore will terminate the job.
>>> --
>>> [pnod0330:110442] [[58323,0],0] orted_cmd: received halt_vm cmd
>>> [pnod0332:56161] [[58323,0],2] orted_cmd: received halt_vm cmd
>>> ```
>>> 
>>> I haven't yet tried the hard reboot case with ULFM (these nodes take
>>> forever to come back up), but earlier experiments SIGKILLing the orted
>>> on a compute node led to a very similar message as above, so at this
>>> point I'm not optimistic...
>>> 
>>> I think my next step is to try with several separate mpiruns and use
>>> mpi_comm_{connect,accept} to plumb everything together before the
>>> application starts.  I notice this is the subject of some recent work
>>> on ompi master.  Even though the mpiruns will all be associated to the
>>> same ompi-server, do you think this could be sufficient to isolate the
>>> failures?
>>> 
>>> Cheers,
>>> Tim
>>> 
>>> 
>>> 
>>> On 10 June 2017 at 00:56, r...@open-mpi.org <r...@open-mpi.org> wrote:
>>>> It

Re: [OMPI users] waiting for message either from MPI communicator or from TCP socket

[r...@open-mpi.org]

I suspect nobody is answering because the question makes no sense to us. You 
are implying that the TCP socket is outside of MPI since it isn’t tied to a 
communicator. If you want to setup a non-MPI communication path between two 
procs and monitor it separate from the MPI library, you can certainly do so - 
there is nothing that prevents you from doing it.

I suspect that question went similarly unanswered for the same reason.

> On Jun 25, 2017, at 11:08 AM, Trevour Spencer  
> wrote:
> 
> Dear all,
> I have not yet found a solution for waiting for an event (incoming message) 
> to occur either on a TCP socket or on a MPI communicator. I wish I could 
> receive some comments from you MPI experts.
> If my question was unclear, I found the same problem described on the MPICH 
> list two years ago: 
> https://lists.mpich.org/pipermail/discuss/2015-June/004049.html 
> 
> He received no reply either.
> 
> Is the question stupid for some reason, is the answer trivial, or is there no 
> solution to this problem?
> 
> Cheers
> Trevour
> 
> 2017-06-20 20:15 GMT+02:00 Trevour Spencer  >:
> Dear all,
> I am looking for a solution to receive messages either via MPI or via a TCP 
> socket, whichever arrives next.
> 
> Using the select() method as described for example at 
> http://developerweb.net/viewtopic.php?id=2933 
> , I can have my code wait 
> until any of a given set of TCP sockets receives some data.
> 
> Does a similar solution exist, such that the code waits until data is 
> available either from an MPI communicator or from a TCP socket?
> 
> Cheers
> Trevour
> 
>  
> 
> Virenfrei. www.avast.com 
> 
>  
> ___
> users mailing list
> users@lists.open-mpi.org
> https://rfd.newmexicoconsortium.org/mailman/listinfo/users

___
users mailing list
users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

Re: [OMPI users] --host works but --hostfile does not

[r...@open-mpi.org]

From “man mpirun” - note that not specifying “slots=N” in a hostfile defaults 
to slots=#cores on that node (as it states in the text):

   Specifying Host Nodes
   Host nodes can be identified on the mpirun command line with the -host 
option or in a hostfile.

   For example,

   mpirun -H aa,aa,bb ./a.out
   launches two processes on node aa and one on bb.

   Or, consider the hostfile

  % cat myhostfile
  aa slots=2
  bb slots=2
  cc slots=2

   Here, we list both the host names (aa, bb, and cc) but also how many 
"slots" there are for each.  Slots indicate how many processes can potentially
   execute on a node.  For best performance, the number of slots may be 
chosen to be the number of cores on the node or the number of processor  sock‐
   ets.   If  the  hostfile  does  not  provide  slots  information,  Open 
MPI will attempt to discover the number of cores (or hwthreads, if the use-
   hwthreads-as-cpus option is set) and set the number of slots to that 
value. This default behavior also occurs when specifying the -host option with
   a single hostname. Thus, the command

   mpirun -H aa ./a.out
   launches a number of processes equal to the number of cores on node 
aa.

   mpirun -hostfile myhostfile ./a.out
   will launch two processes on each of the three nodes.

   mpirun -hostfile myhostfile -host aa ./a.out
   will launch two processes, both on node aa.

   mpirun -hostfile myhostfile -host dd ./a.out
   will find no hosts to run on and abort with an error.  That is, the 
specified host dd is not in the specified hostfile.

   When running under resource managers (e.g., SLURM, Torque, etc.), Open 
MPI will obtain both the hostnames and the number of slots directly from the
   resource manger.

   Specifying Number of Processes
   As we have just seen, the number of processes to run can be set using 
the hostfile.  Other mechanisms exist.

   The number of processes launched can be specified as a multiple of the 
number of nodes or processor sockets available.  For example,

   mpirun -H aa,bb -npersocket 2 ./a.out
   launches processes 0-3 on node aa and process 4-7 on node bb, where 
aa and bb are both dual-socket nodes.  The -npersocket option also turns on
   the -bind-to-socket option, which is discussed in a later section.

   mpirun -H aa,bb -npernode 2 ./a.out
   launches processes 0-1 on node aa and processes 2-3 on node bb.

   mpirun -H aa,bb -npernode 1 ./a.out
   launches one process per host node.

   mpirun -H aa,bb -pernode ./a.out
   is the same as -npernode 1.

   Another alternative is to specify the number of processes with the -np 
option.  Consider now the hostfile

  % cat myhostfile
  aa slots=4
  bb slots=4
  cc slots=4

   Now,

   mpirun -hostfile myhostfile -np 6 ./a.out
   will  launch  processes 0-3 on node aa and processes 4-5 on node bb. 
 The remaining slots in the hostfile will not be used since the -np option
   indicated that only 6 processes should be launched.


> On Jun 22, 2017, at 7:49 AM, Info via users  wrote:
> 
> I am just learning to use openmpi 1.8.4 that is installed on our cluster. I 
> am running into a baffling issue. If I run:
> 
> mpirun -np 3 --host b1,b2,b3 hostname
> 
> I get the expected output:
> 
> b1
> b2
> b3
> 
> But if I do:
> 
> mpirun -np 3 --hostfile hostfile hostname
> 
> I get:
> 
> b1
> b1
> b1
> 
> Where hostfile contains:
> 
> b1
> b2
> b3
> 
> Any ideas what could going on?
> ___
> users mailing list
> users@lists.open-mpi.org
> https://rfd.newmexicoconsortium.org/mailman/listinfo/users

___
users mailing list
users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

Re: [OMPI users] disable slurm/munge from mpirun

[r...@open-mpi.org]

I gather you are using OMPI 2.x, yes? And you configured it 
--with-pmi=, then moved the executables/libs to your workstation?

I suppose I could state the obvious and say “don’t do that - just rebuild it”, 
and I fear that (after checking the 2.x code) you really have no choice. OMPI 
v3.0 will have a way around the problem, but not the 2.x series.


> On Jun 22, 2017, at 8:04 AM, Michael Di Domenico  
> wrote:
> 
> On Thu, Jun 22, 2017 at 10:43 AM, John Hearns via users
>  wrote:
>> Having had some problems with ssh launching (a few minutes ago) I can
>> confirm that this works:
>> 
>> --mca plm_rsh_agent "ssh -v"
> 
> this doesn't do anything for me
> 
> if i set OMPI_MCA_sec=^munge
> 
> i can clear the mca_sec_munge error
> 
> but the mca_pmix_pmix112 and opal_pmix_base_select errors still
> exists.  the plm_rsh_agent switch/env var doesn't seem to affect that
> error
> 
> down the road, i may still need the rsh_agent flag, but i think we're
> still before that in the sequence of events
> ___
> users mailing list
> users@lists.open-mpi.org
> https://rfd.newmexicoconsortium.org/mailman/listinfo/users

___
users mailing list
users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/users