Re: [OMPI users] mpi4py+OpenMPI: Qs about submitting bugs and examples

[r...@open-mpi.org]

> On Oct 31, 2016, at 10:39 AM, Jason Maldonis  wrote:
> 
> Hello everyone,
> 
> I am using mpi4py with OpenMPI for a simulation that uses dynamic resource 
> allocation via `mpi_spawn_multiple`.  I've been working on this problem for 
> about 6 months now and I have some questions and potential bugs I'd like to 
> submit.
> 
> Is this mailing list a good spot to submit bugs for OpenMPI? Or do I use 
> github?

You can use either - I would encourage the use of github “issues” when you have 
a specific bug, and the mailing list for general questions

> Are previous versions (like 1.10.2) still being developed for bugfixes, or do 
> I need to reproduce bugs for 2.x only?

The 1.10 series is still being supported - it has proven fairly stable and so 
the release rate has slowed down considerably in the last year. Primary 
development focus in on 2.x

> 
> I may also submit bugs to mpi4py, but I don't yet know exactly where the bugs 
> are originating from.  Do any of you know if github is the correct place to 
> submit bugs for mpi4py?

I honestly don’t know, but I do believe mpi4py is on github as well

> 
> I have also learned some cool things that are not well documented on the web, 
> and I'd like to provide nice examples or something similar. Can I contribute 
> examples to either mpi4py or OpenMPI?

Please do!

> 
> As a side note, OpenMPI 1.10.2 seems to be much more stable than 2.x for the 
> dynamic resource allocation code I am writing.

Yes, there has been an outstanding bug on the 2.x series for dynamic 
operations. We just finally found the missing code change and it is being 
ported at this time.

> 
> Thanks in advance,
> Jason Maldonis
> 
> ___
> users mailing list
> users@lists.open-mpi.org
> https://rfd.newmexicoconsortium.org/mailman/listinfo/users

___
users mailing list
users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

Re: [OMPI users] what was the rationale behind rank mapping by socket?

[r...@open-mpi.org]

ernatively,  processes mapped by l2cache and then bound to socket 
will simply be bound to all the processors in the socket where they are 
located. In this manner,
   users can exert detailed control over relative MCW rank location and 
binding.

   Finally, --report-bindings can be used to report bindings.

   As an example, consider a node with two processor sockets, each 
comprising four cores.  We run mpirun with -np  4  --report-bindings  and  the  
following  additional
   options:

% mpirun ... --map-by core --bind-to core
[...] ... binding child [...,0] to cpus 0001
[...] ... binding child [...,1] to cpus 0002
[...] ... binding child [...,2] to cpus 0004
[...] ... binding child [...,3] to cpus 0008

% mpirun ... --map-by socket --bind-to socket
[...] ... binding child [...,0] to socket 0 cpus 000f
[...] ... binding child [...,1] to socket 1 cpus 00f0
[...] ... binding child [...,2] to socket 0 cpus 000f
[...] ... binding child [...,3] to socket 1 cpus 00f0

% mpirun ... --map-by core:PE=2 --bind-to core
[...] ... binding child [...,0] to cpus 0003
[...] ... binding child [...,1] to cpus 000c
[...] ... binding child [...,2] to cpus 0030
[...] ... binding child [...,3] to cpus 00c0

% mpirun ... --bind-to none


  Here, --report-bindings shows the binding of each process as a mask.  In 
the first case, the processes bind to successive cores as indicated by the 
masks 0001, 0002,
   0004, and 0008.  In the second case, processes bind to all cores on 
successive sockets as indicated by the masks 000f and 00f0.  The processes 
cycle through the pro‐
   cessor  sockets  in a round-robin fashion as many times as are needed.  
In the third case, the masks show us that 2 cores have been bound per process.  
In the fourth
   case, binding is turned off and no bindings are reported.

   Open MPI's support for process binding depends on the underlying 
operating system.  Therefore, certain process binding options may not be 
available on every system.

   Process binding can also be set with MCA parameters.  Their usage is 
less convenient than that of mpirun options.  On the other hand, MCA parameters 
can be  set  not
   only on the mpirun command line, but alternatively in a system or user 
mca-params.conf file or as environment variables, as described in the MCA 
section below.  Some
   examples include:

   mpirun option  MCA parameter key value

 --map-by core  rmaps_base_mapping_policy   core
 --map-by socketrmaps_base_mapping_policy   socket
 --rank-by core rmaps_base_ranking_policy   core
 --bind-to core hwloc_base_binding_policy   core
 --bind-to socket   hwloc_base_binding_policy   socket
 --bind-to none hwloc_base_binding_policy   none


> On Oct 28, 2016, at 4:50 PM, Bennet Fauber <ben...@umich.edu> wrote:
> 
> Ralph,
> 
> Alas, I will not be at SC16.  I would like to hear and/or see what you
> present, so if it gets made available in alternate format, I'd
> appreciated know where and how to get it.
> 
> I am more and more coming to think that our cluster configuration is
> essentially designed to frustrated MPI developers because we use the
> scheduler to create cgroups (once upon a time, cpusets) for subsets of
> cores on multisocket machines, and I think that invalidates a lot of
> the assumptions that are getting made by people who want to bind to
> particular patters.
> 
> It's our foot, and we have been doing a good job of shooting it.  ;-)
> 
> -- bennet
> 
> 
> 
> 
> On Fri, Oct 28, 2016 at 7:18 PM, r...@open-mpi.org <r...@open-mpi.org> wrote:
>> FWIW: I’ll be presenting “Mapping, Ranking, and Binding - Oh My!” at the
>> OMPI BoF meeting at SC’16, for those who can attend. Will try to explain the
>> rationale as well as the mechanics of the options
>> 
>> On Oct 11, 2016, at 8:09 AM, Dave Love <d.l...@liverpool.ac.uk> wrote:
>> 
>> Gilles Gouaillardet <gil...@rist.or.jp> writes:
>> 
>> Bennet,
>> 
>> 
>> my guess is mapping/binding to sockets was deemed the best compromise
>> from an
>> 
>> "out of the box" performance point of view.
>> 
>> 
>> iirc, we did fix some bugs that occured when running under asymmetric
>> cpusets/cgroups.
>> 
>> if you still have some issues with the latest Open MPI version (2.0.1)
>> and the default policy,
>> 
>> could you please describe them ?
>> 
>> 
>> I also don't understand why binding to sockets is the right thing to do.
>> Binding to cores seems the right default to me, and I set that locally,

Re: [OMPI users] what was the rationale behind rank mapping by socket?

[r...@open-mpi.org]

FWIW: I’ll be presenting “Mapping, Ranking, and Binding - Oh My!” at the OMPI 
BoF meeting at SC’16, for those who can attend. Will try to explain the 
rationale as well as the mechanics of the options

> On Oct 11, 2016, at 8:09 AM, Dave Love  wrote:
> 
> Gilles Gouaillardet > writes:
> 
>> Bennet,
>> 
>> 
>> my guess is mapping/binding to sockets was deemed the best compromise
>> from an
>> 
>> "out of the box" performance point of view.
>> 
>> 
>> iirc, we did fix some bugs that occured when running under asymmetric
>> cpusets/cgroups.
>> 
>> if you still have some issues with the latest Open MPI version (2.0.1)
>> and the default policy,
>> 
>> could you please describe them ?
> 
> I also don't understand why binding to sockets is the right thing to do.
> Binding to cores seems the right default to me, and I set that locally,
> with instructions about running OpenMP.  (Isn't that what other
> implementations do, which makes them look better?)
> 
> I think at least numa should be used, rather than socket.  Knights
> Landing, for instance, is single-socket, so no gets no actual binding by
> default.
> ___
> users mailing list
> users@lists.open-mpi.org 
> https://rfd.newmexicoconsortium.org/mailman/listinfo/users 
> 
___
users mailing list
users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

Re: [OMPI users] Launching hybrid MPI/OpenMP jobs on a cluster: correct OpenMPI flags?

[r...@open-mpi.org]

FWIW: I’ll be presenting “Mapping, Ranking, and Binding - Oh My!” at the OMPI 
BoF meeting at SC’16, for those who can attend


> On Oct 11, 2016, at 8:16 AM, Dave Love  wrote:
> 
> Wirawan Purwanto  writes:
> 
>> Instead of the scenario above, I was trying to get the MPI processes
>> side-by-side (more like "fill_up" policy in SGE scheduler), i.e. fill
>> node 0 first, then fill node 1, and so on. How do I do this properly?
>> 
>> I tried a few attempts that fail:
>> 
>> $ export OMP_NUM_THREADS=2
>> $ mpirun -np 16 -map-by core:PE=2 ./EXECUTABLE
> 
> ...
> 
>> Clearly I am not understanding how this map-by works. Could somebody
>> help me? There was a wiki article partially written:
>> 
>> https://github.com/open-mpi/ompi/wiki/ProcessPlacement
>> 
>> but unfortunately it is also not clear to me.
> 
> Me neither; this stuff has traditionally been quite unclear and really
> needs documenting/explaining properly.
> 
> This sort of thing from my local instructions for OMPI 1.8 probably does
> what you want for OMP_NUM_THREADS=2 (where the qrsh options just get me
> a couple of small nodes):
> 
>  $ qrsh -pe mpi 24 -l num_proc=12 \
> mpirun -n 12 --map-by slot:PE=2 --bind-to core --report-bindings true |&
> sort -k 4 -n
>  [comp544:03093] MCW rank 0 bound to socket 0[core 0[hwt 0]], socket 0[core 
> 1[hwt 0]]: [B/B/./././.][./././././.]
>  [comp544:03093] MCW rank 1 bound to socket 0[core 2[hwt 0]], socket 0[core 
> 3[hwt 0]]: [././B/B/./.][./././././.]
>  [comp544:03093] MCW rank 2 bound to socket 0[core 4[hwt 0]], socket 0[core 
> 5[hwt 0]]: [././././B/B][./././././.]
>  [comp544:03093] MCW rank 3 bound to socket 1[core 6[hwt 0]], socket 1[core 
> 7[hwt 0]]: [./././././.][B/B/./././.]
>  [comp544:03093] MCW rank 4 bound to socket 1[core 8[hwt 0]], socket 1[core 
> 9[hwt 0]]: [./././././.][././B/B/./.]
>  [comp544:03093] MCW rank 5 bound to socket 1[core 10[hwt 0]], socket 1[core 
> 11[hwt 0]]: [./././././.][././././B/B]
>  [comp527:03056] MCW rank 6 bound to socket 0[core 0[hwt 0]], socket 0[core 
> 1[hwt 0]]: [B/B/./././.][./././././.]
>  [comp527:03056] MCW rank 7 bound to socket 0[core 2[hwt 0]], socket 0[core 
> 3[hwt 0]]: [././B/B/./.][./././././.]
>  [comp527:03056] MCW rank 8 bound to socket 0[core 4[hwt 0]], socket 0[core 
> 5[hwt 0]]: [././././B/B][./././././.]
>  [comp527:03056] MCW rank 9 bound to socket 1[core 6[hwt 0]], socket 1[core 
> 7[hwt 0]]: [./././././.][B/B/./././.]
>  [comp527:03056] MCW rank 10 bound to socket 1[core 8[hwt 0]], socket 1[core 
> 9[hwt 0]]: [./././././.][././B/B/./.]
>  [comp527:03056] MCW rank 11 bound to socket 1[core 10[hwt 0]], socket 1[core 
> 11[hwt 0]]: [./././././.][././././B/B]
> 
> I don't remember how I found that out.
> ___
> users mailing list
> users@lists.open-mpi.org
> https://rfd.newmexicoconsortium.org/mailman/listinfo/users

___
users mailing list
users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

Re: [OMPI users] MCA compilation later

[r...@open-mpi.org]

You don’t need any of the hardware - you just need the headers. Things like 
libfabric and libibverbs are all publicly available, and so you can build all 
that support even if you cannot run it on your machine.

Once your customer installs the binary, the various plugins will check for 
their required library and hardware and disqualify themselves if it isn’t found.

> On Oct 28, 2016, at 12:33 PM, Sean Ahern  wrote:
> 
> There's been discussion on the OpenMPI list recently about static linking of 
> OpenMPI with all of the desired MCAs in it. I've got the opposite question. 
> I'd like to add MCAs later on to an already-compiled version of OpenMPI and 
> am not quite sure how to do it.
> 
> Let me summarize. We've got a commercial code that we deploy on customer 
> machines in binary form. We're working to integrate OpenMPI into the 
> installer, and things seem to be progressing well. (Note: because we're a 
> commercial code, making the customer compile something doesn't work for us 
> like it can for open source or research codes.)
> 
> Now, we want to take advantage of OpenMPI's ability to find MCAs at runtime, 
> pointing to the various plugins that might apply to a deployed system. I've 
> configured and compiled OpenMPI on one of our build machines, one that 
> doesn't have any special interconnect hardware or software installed. We take 
> this compiled version of OpenMPI and use it on all of our machines. (Yes, 
> I've read Building FAQ #39 
>  about 
> relocating OpenMPI. Useful, that.) I'd like to take our pre-compiled version 
> of OpenMPI and add MCA libraries to it, giving OpenMPI the ability to 
> communicate via transport mechanisms that weren't available on the original 
> build machine. Things like InfiniBand, OmniPath, or one of Cray's 
> interconnects.
> 
> How would I go about doing this? And what are the limitations?
> 
> I'm guessing that I need to go configure and compile the same version of 
> OpenMPI on a machine that has the desired interconnect installation (headers 
> and libraries), then go grab the corresponding lib/openmpi/mca_*{la,so} 
> files. Take those files and drop them in our pre-built OpenMPI from our build 
> machine in the same relative plugin location (lib/openmpi). If I stick with 
> the same compiler (gcc, in this case), I'm hoping that symbols will all 
> resolve themselves at runtime. (I probably will have to do some 
> LD_LIBRARY_PATH games to be sure to find the appropriate underlying libraries 
> unless OpenMPI's process for building MCAs links them in statically somehow.)
> 
> Am I even on the right track here? (The various system-level FAQs (here 
> , here 
> , and especially here 
> ) seem to suggest that I am.)
> 
> Our first test platform will be getting OpenMPI via IB working on our 
> cluster, where we have IB (and TCP/IP) functional and not OpenMPI. This will 
> be a great stand-in for a customer that has an IB cluster and wants to just 
> run our binary installation.
> 
> Thanks.
> 
> -Sean
> 
> --
> Sean Ahern
> Computational Engineering International
> 919-363-0883
> ___
> users mailing list
> users@lists.open-mpi.org
> https://rfd.newmexicoconsortium.org/mailman/listinfo/users

___
users mailing list
users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

Re: [OMPI users] Slurm binding not propagated to MPI jobs

[r...@open-mpi.org]

Sigh - of course it wouldn’t be simple :-(

All right, let’s suppose we look for SLURM_CPU_BIND:

* if it includes the word “none”, then we know the user specified that they 
don’t want us to bind

* if it includes the word mask_cpu, then we have to check the value of that 
option.

* If it is all F’s, then they didn’t specify a binding and we should do our 
thing.

* If it is anything else, then we assume they _did_ specify a binding, and we 
leave it alone

Would that make sense? Is there anything else that could be in that envar which 
would trip us up?


> On Oct 27, 2016, at 10:37 AM, Andy Riebs <andy.ri...@hpe.com> wrote:
> 
> Yes, they still exist:
> 
> $ srun --ntasks-per-node=2 -N1 env | grep BIND | sort -u
> SLURM_CPU_BIND_LIST=0x
> SLURM_CPU_BIND=quiet,mask_cpu:0x
> SLURM_CPU_BIND_TYPE=mask_cpu:
> SLURM_CPU_BIND_VERBOSE=quiet
> Here are the relevant Slurm configuration options that could conceivably 
> change the behavior from system to system:
> SelectType  = select/cons_res
> SelectTypeParameters    = CR_CPU
> 
> 
> On 10/27/2016 01:17 PM, r...@open-mpi.org <mailto:r...@open-mpi.org> wrote:
>> And if there is no --cpu_bind on the cmd line? Do these not exist?
>> 
>>> On Oct 27, 2016, at 10:14 AM, Andy Riebs <andy.ri...@hpe.com 
>>> <mailto:andy.ri...@hpe.com>> wrote:
>>> 
>>> Hi Ralph,
>>> 
>>> I think I've found the magic keys...
>>> 
>>> $ srun --ntasks-per-node=2 -N1 --cpu_bind=none env | grep BIND
>>> SLURM_CPU_BIND_VERBOSE=quiet
>>> SLURM_CPU_BIND_TYPE=none
>>> SLURM_CPU_BIND_LIST=
>>> SLURM_CPU_BIND=quiet,none
>>> SLURM_CPU_BIND_VERBOSE=quiet
>>> SLURM_CPU_BIND_TYPE=none
>>> SLURM_CPU_BIND_LIST=
>>> SLURM_CPU_BIND=quiet,none
>>> $ srun --ntasks-per-node=2 -N1 --cpu_bind=core env | grep BIND
>>> SLURM_CPU_BIND_VERBOSE=quiet
>>> SLURM_CPU_BIND_TYPE=mask_cpu:
>>> SLURM_CPU_BIND_LIST=0x1111,0x
>>> SLURM_CPU_BIND=quiet,mask_cpu:0x,0x
>>> SLURM_CPU_BIND_VERBOSE=quiet
>>> SLURM_CPU_BIND_TYPE=mask_cpu:
>>> SLURM_CPU_BIND_LIST=0x,0x
>>> SLURM_CPU_BIND=quiet,mask_cpu:0x,0x
>>> 
>>> Andy
>>> 
>>> On 10/27/2016 11:57 AM, r...@open-mpi.org <mailto:r...@open-mpi.org> wrote:
>>>> Hey Andy
>>>> 
>>>> Is there a SLURM envar that would tell us the binding option from the srun 
>>>> cmd line? We automatically bind when direct launched due to user 
>>>> complaints of poor performance if we don’t. If the user specifies a 
>>>> binding option, then we detect that we were already bound and don’t 
>>>> do it.
>>>> 
>>>> However, if the user specifies that they not be bound, then we think they 
>>>> simply didn’t specify anything - and that isn’t the case. If 
>>>> we can see something that tells us “they explicitly said not to do 
>>>> it”, then we can avoid the situation.
>>>> 
>>>> Ralph
>>>> 
>>>>> On Oct 27, 2016, at 8:48 AM, Andy Riebs <andy.ri...@hpe.com 
>>>>> <mailto:andy.ri...@hpe.com>> wrote:
>>>>> 
>>>>> Hi All,
>>>>> 
>>>>> We are running Open MPI version 1.10.2, built with support for Slurm 
>>>>> version 16.05.0. When a user specifies "--cpu_bind=none", MPI tries to 
>>>>> bind by core, which segv's if there are more processes than cores.
>>>>> 
>>>>> The user reports:
>>>>> 
>>>>> What I found is that
>>>>> 
>>>>> % srun --ntasks-per-node=8 --cpu_bind=none  \
>>>>> env SHMEM_SYMMETRIC_HEAP_SIZE=1024M bin/all2all.shmem.exe 0
>>>>> 
>>>>> will have the problem, but:
>>>>> 
>>>>> % srun --ntasks-per-node=8 --cpu_bind=none  \
>>>>> env SHMEM_SYMMETRIC_HEAP_SIZE=1024M ./bindit.sh bin/all2all.shmem.exe >>>>> 0
>>>>> 
>>>>> Will run as expected and print out the usage message because I 
>>>>> didn’t provide the right arguments to the code.
>>>>> 
>>>>> So, it appears that the binding has something to do with the issue. My 
>>>>> binding script is as follows:
>>>>> 
>>>>> % cat bindit.sh
>>>>> #!/bin/bash
>>>>> 
>>>>> #echo SLURM_LOCALID=$SLURM_LOCALID
>>>>> 
&g

Re: [OMPI users] Slurm binding not propagated to MPI jobs

[r...@open-mpi.org]

And if there is no --cpu_bind on the cmd line? Do these not exist?

> On Oct 27, 2016, at 10:14 AM, Andy Riebs <andy.ri...@hpe.com> wrote:
> 
> Hi Ralph,
> 
> I think I've found the magic keys...
> 
> $ srun --ntasks-per-node=2 -N1 --cpu_bind=none env | grep BIND
> SLURM_CPU_BIND_VERBOSE=quiet
> SLURM_CPU_BIND_TYPE=none
> SLURM_CPU_BIND_LIST=
> SLURM_CPU_BIND=quiet,none
> SLURM_CPU_BIND_VERBOSE=quiet
> SLURM_CPU_BIND_TYPE=none
> SLURM_CPU_BIND_LIST=
> SLURM_CPU_BIND=quiet,none
> $ srun --ntasks-per-node=2 -N1 --cpu_bind=core env | grep BIND
> SLURM_CPU_BIND_VERBOSE=quiet
> SLURM_CPU_BIND_TYPE=mask_cpu:
> SLURM_CPU_BIND_LIST=0x,0x
> SLURM_CPU_BIND=quiet,mask_cpu:0x,0x
> SLURM_CPU_BIND_VERBOSE=quiet
> SLURM_CPU_BIND_TYPE=mask_cpu:
> SLURM_CPU_BIND_LIST=0x,0x
> SLURM_CPU_BIND=quiet,mask_cpu:0x,0x
> 
> Andy
> 
> On 10/27/2016 11:57 AM, r...@open-mpi.org <mailto:r...@open-mpi.org> wrote:
>> Hey Andy
>> 
>> Is there a SLURM envar that would tell us the binding option from the srun 
>> cmd line? We automatically bind when direct launched due to user complaints 
>> of poor performance if we don’t. If the user specifies a binding option, 
>> then we detect that we were already bound and don’t do it.
>> 
>> However, if the user specifies that they not be bound, then we think they 
>> simply didn’t specify anything - and that isn’t the case. If we can see 
>> something that tells us “they explicitly said not to do it”, then we can 
>> avoid the situation.
>> 
>> Ralph
>> 
>>> On Oct 27, 2016, at 8:48 AM, Andy Riebs <andy.ri...@hpe.com 
>>> <mailto:andy.ri...@hpe.com>> wrote:
>>> 
>>> Hi All,
>>> 
>>> We are running Open MPI version 1.10.2, built with support for Slurm 
>>> version 16.05.0. When a user specifies "--cpu_bind=none", MPI tries to bind 
>>> by core, which segv's if there are more processes than cores.
>>> 
>>> The user reports:
>>> 
>>> What I found is that
>>> 
>>> % srun --ntasks-per-node=8 --cpu_bind=none  \
>>> env SHMEM_SYMMETRIC_HEAP_SIZE=1024M bin/all2all.shmem.exe 0
>>> 
>>> will have the problem, but:
>>> 
>>> % srun --ntasks-per-node=8 --cpu_bind=none  \
>>> env SHMEM_SYMMETRIC_HEAP_SIZE=1024M ./bindit.sh bin/all2all.shmem.exe 0
>>> 
>>> Will run as expected and print out the usage message because I didn’t 
>>> provide the right arguments to the code.
>>> 
>>> So, it appears that the binding has something to do with the issue. My 
>>> binding script is as follows:
>>> 
>>> % cat bindit.sh
>>> #!/bin/bash
>>> 
>>> #echo SLURM_LOCALID=$SLURM_LOCALID
>>> 
>>> stride=1
>>> 
>>> if [ ! -z "$SLURM_LOCALID" ]; then
>>>   let bindCPU=$SLURM_LOCALID*$stride
>>>   exec numactl --membind=0 --physcpubind=$bindCPU $*
>>> fi
>>> 
>>> $*
>>> 
>>> %
>>> 
>>> 
>>> -- 
>>> Andy Riebs
>>> andy.ri...@hpe.com
>>> Hewlett-Packard Enterprise
>>> High Performance Computing Software Engineering
>>> +1 404 648 9024
>>> My opinions are not necessarily those of HPE
>>>May the source be with you!
>>> 
>>> ___
>>> users mailing list
>>> users@lists.open-mpi.org
>>> https://rfd.newmexicoconsortium.org/mailman/listinfo/users
>> ___
>> users mailing list
>> users@lists.open-mpi.org
>> https://rfd.newmexicoconsortium.org/mailman/listinfo/users
> 
> ___
> users mailing list
> users@lists.open-mpi.org <mailto:users@lists.open-mpi.org>
> https://rfd.newmexicoconsortium.org/mailman/listinfo/users 
> <https://rfd.newmexicoconsortium.org/mailman/listinfo/users>
___
users mailing list
users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

Re: [OMPI users] Slurm binding not propagated to MPI jobs

[r...@open-mpi.org]

Hey Andy

Is there a SLURM envar that would tell us the binding option from the srun cmd 
line? We automatically bind when direct launched due to user complaints of poor 
performance if we don’t. If the user specifies a binding option, then we detect 
that we were already bound and don’t do it.

However, if the user specifies that they not be bound, then we think they 
simply didn’t specify anything - and that isn’t the case. If we can see 
something that tells us “they explicitly said not to do it”, then we can avoid 
the situation.

Ralph

> On Oct 27, 2016, at 8:48 AM, Andy Riebs  wrote:
> 
> Hi All,
> 
> We are running Open MPI version 1.10.2, built with support for Slurm version 
> 16.05.0. When a user specifies "--cpu_bind=none", MPI tries to bind by core, 
> which segv's if there are more processes than cores.
> 
> The user reports:
> 
> What I found is that
> 
> % srun --ntasks-per-node=8 --cpu_bind=none  \
> env SHMEM_SYMMETRIC_HEAP_SIZE=1024M bin/all2all.shmem.exe 0
> 
> will have the problem, but:
> 
> % srun --ntasks-per-node=8 --cpu_bind=none  \
> env SHMEM_SYMMETRIC_HEAP_SIZE=1024M ./bindit.sh bin/all2all.shmem.exe 0
> 
> Will run as expected and print out the usage message because I didn’t provide 
> the right arguments to the code.
> 
> So, it appears that the binding has something to do with the issue. My 
> binding script is as follows:
> 
> % cat bindit.sh
> #!/bin/bash
> 
> #echo SLURM_LOCALID=$SLURM_LOCALID
> 
> stride=1
> 
> if [ ! -z "$SLURM_LOCALID" ]; then
>   let bindCPU=$SLURM_LOCALID*$stride
>   exec numactl --membind=0 --physcpubind=$bindCPU $*
> fi
> 
> $*
> 
> %
> 
> 
> -- 
> Andy Riebs
> andy.ri...@hpe.com
> Hewlett-Packard Enterprise
> High Performance Computing Software Engineering
> +1 404 648 9024
> My opinions are not necessarily those of HPE
>May the source be with you!
> 
> ___
> users mailing list
> users@lists.open-mpi.org
> https://rfd.newmexicoconsortium.org/mailman/listinfo/users

___
users mailing list
users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

[OMPI users] Supercomputing 2016: Birds-of-a-Feather meetings

[r...@open-mpi.org]

Hello all

This year, we will again be hosting Birds-of-a-Feather meetings for Open MPI 
and PMIx. 

Open MPI: Wed, Nov 16th, 5:15-7pm

http://sc16.supercomputing.org/presentation/?id=bof103=sess322 



PMIx: Wed, Nov16th, 12:15-1:15pm:

http://sc16.supercomputing.org/presentation/?id=bof104=sess323 


Please plan to attend!

Ralph

___
users mailing list
users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

Re: [OMPI users] how to tell if pmi or pmi2 is being used?

[r...@open-mpi.org]

If you are using mpirun, then neither PMI1 or PMI2 are involved at all. ORTE 
has its own internal mechanism for handling wireup.


> On Oct 13, 2016, at 10:43 AM, David Shrader  wrote:
> 
> Hello All,
> 
> I'm using Open MPI 1.10.3 with Slurm and would like to ask how do I find out 
> if pmi1 or pmi2 was used for process launching? The Slurm installation is 
> supposed to support both pmi1 and pmi2, but I would really like to know which 
> one I fall in to. I tried using '-mca plm_base_verbose 100' on the mpirun 
> line, but it didn't mention pmi specifically. Instead, all I could really 
> find was that it was using the slurm component. Is there something else I can 
> look at in the output that would have that detail?
> 
> Thank you for your time,
> David
> 
> -- 
> David Shrader
> HPC-ENV High Performance Computer Systems
> Los Alamos National Lab
> Email: dshrader  lanl.gov
> 
> ___
> users mailing list
> users@lists.open-mpi.org
> https://rfd.newmexicoconsortium.org/mailman/listinfo/users

___
users mailing list
users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

Re: [OMPI users] Launching hybrid MPI/OpenMP jobs on a cluster: correct OpenMPI flags?

[r...@open-mpi.org]

FWIW: the socket option seems to work fine for me:

$ mpirun -n 12 -map-by socket:pe=2 -host rhc001 --report-bindings hostname
[rhc001:200408] MCW rank 1 bound to socket 1[core 12[hwt 0-1]], socket 1[core 
13[hwt 0-1]]: 
[../../../../../../../../../../../..][BB/BB/../../../../../../../../../..]
[rhc001:200408] MCW rank 2 bound to socket 0[core 2[hwt 0-1]], socket 0[core 
3[hwt 0-1]]: 
[../../BB/BB/../../../../../../../..][../../../../../../../../../../../..]
[rhc001:200408] MCW rank 3 bound to socket 1[core 14[hwt 0-1]], socket 1[core 
15[hwt 0-1]]: 
[../../../../../../../../../../../..][../../BB/BB/../../../../../../../..]
[rhc001:200408] MCW rank 4 bound to socket 0[core 4[hwt 0-1]], socket 0[core 
5[hwt 0-1]]: 
[../../../../BB/BB/../../../../../..][../../../../../../../../../../../..]
[rhc001:200408] MCW rank 5 bound to socket 1[core 16[hwt 0-1]], socket 1[core 
17[hwt 0-1]]: 
[../../../../../../../../../../../..][../../../../BB/BB/../../../../../..]
[rhc001:200408] MCW rank 6 bound to socket 0[core 6[hwt 0-1]], socket 0[core 
7[hwt 0-1]]: 
[../../../../../../BB/BB/../../../..][../../../../../../../../../../../..]
[rhc001:200408] MCW rank 7 bound to socket 1[core 18[hwt 0-1]], socket 1[core 
19[hwt 0-1]]: 
[../../../../../../../../../../../..][../../../../../../BB/BB/../../../..]
[rhc001:200408] MCW rank 8 bound to socket 0[core 8[hwt 0-1]], socket 0[core 
9[hwt 0-1]]: 
[../../../../../../../../BB/BB/../..][../../../../../../../../../../../..]
[rhc001:200408] MCW rank 9 bound to socket 1[core 20[hwt 0-1]], socket 1[core 
21[hwt 0-1]]: 
[../../../../../../../../../../../..][../../../../../../../../BB/BB/../..]
[rhc001:200408] MCW rank 10 bound to socket 0[core 10[hwt 0-1]], socket 0[core 
11[hwt 0-1]]: 
[../../../../../../../../../../BB/BB][../../../../../../../../../../../..]
[rhc001:200408] MCW rank 11 bound to socket 1[core 22[hwt 0-1]], socket 1[core 
23[hwt 0-1]]: 
[../../../../../../../../../../../..][../../../../../../../../../../BB/BB]
[rhc001:200408] MCW rank 0 bound to socket 0[core 0[hwt 0-1]], socket 0[core 
1[hwt 0-1]]: 
[BB/BB/../../../../../../../../../..][../../../../../../../../../../../..]
rhc001
rhc001
rhc001
rhc001
rhc001
rhc001
rhc001
rhc001
rhc001
rhc001
rhc001
rhc001
$

I know that isn’t the pattern you are seeking - will have to ponder that one a 
bit. Is it possible that mpirun is not sitting on the same topology as your 
compute nodes?


> On Oct 3, 2016, at 2:22 PM, Wirawan Purwanto  wrote:
> 
> Hi,
> 
> I have been trying to understand how to correctly launch hybrid
> MPI/OpenMP (i.e. multi-threaded MPI jobs) with mpirun. I am quite
> puzzled as to what is the correct command-line options to use. The
> description on mpirun man page is very confusing and I could not get
> what I wanted.
> 
> A background: The cluster is using SGE, and I am using OpenMPI 1.10.2
> compiled with & for gcc 4.9.3. The MPI library was configured with SGE
> support. The compute nodes have 32 cores, which are basically 2
> sockets of Xeon E5-2698 v3 (16-core Haswell).
> 
> A colleague told me the following:
> 
> $ export OMP_NUM_THREADS=2
> $ mpirun -np 16 -map-by node:PE=2 ./EXECUTABLE
> 
> I could see the executable using 200% of CPU per process--that's good.
> There is one catch in the general case. "-map-by node" will assign the
> MPI processes in a round-robin fashion (so MPI rank 0 gets node 0, mpi
> rank 1 gets node 1, and so on until all nodes are given 1 process,
> then it will go back to node 0,1, ...).
> 
> Instead of the scenario above, I was trying to get the MPI processes
> side-by-side (more like "fill_up" policy in SGE scheduler), i.e. fill
> node 0 first, then fill node 1, and so on. How do I do this properly?
> 
> I tried a few attempts that fail:
> 
> $ export OMP_NUM_THREADS=2
> $ mpirun -np 16 -map-by core:PE=2 ./EXECUTABLE
> 
> or
> 
> $ export OMP_NUM_THREADS=2
> $ mpirun -np 16 -map-by socket:PE=2 ./EXECUTABLE
> 
> Both failed with an error mesage:
> 
> --
> A request for multiple cpus-per-proc was given, but a directive
> was also give to map to an object level that cannot support that
> directive.
> 
> Please specify a mapping level that has more than one cpu, or
> else let us define a default mapping that will allow multiple
> cpus-per-proc.
> --
> 
> Another attempt was:
> 
> $ export OMP_NUM_THREADS=2
> $ mpirun -np 16 -map-by socket:PE=2 -bind-to socket ./EXECUTABLE
> 
> Here's the error message:
> 
> --
> A request for multiple cpus-per-proc was given, but a conflicting binding
> policy was specified:
> 
>  #cpus-per-proc:  2
>  type of cpus:cores as cpus
>  binding policy given: SOCKET
> 
> The correct binding policy for the given type of cpu is:
> 
>  correct binding policy:  bind-to core
> 
> This is the binding policy we would 

Re: [OMPI users] MPI_Comm_spawn

[r...@open-mpi.org]

Ah, that may be why it wouldn’t show up in the OMPI code base itself. If that 
is the case here, then no - OMPI v2.0.1 does not support comm_spawn for PSM. It 
is fixed in the upcoming 2.0.2

> On Sep 29, 2016, at 6:58 AM, Gilles Gouaillardet 
> <gilles.gouaillar...@gmail.com> wrote:
> 
> Ralph,
> 
> My guess is that ptl.c comes from PSM lib ...
> 
> Cheers,
> 
> Gilles
> 
> On Thursday, September 29, 2016, r...@open-mpi.org <mailto:r...@open-mpi.org> 
> <r...@open-mpi.org <mailto:r...@open-mpi.org>> wrote:
> Spawn definitely does not work with srun. I don’t recognize the name of the 
> file that segfaulted - what is “ptl.c”? Is that in your manager program?
> 
> 
>> On Sep 29, 2016, at 6:06 AM, Gilles Gouaillardet 
>> <gilles.gouaillar...@gmail.com 
>> <javascript:_e(%7B%7D,'cvml','gilles.gouaillar...@gmail.com');>> wrote:
>> 
>> Hi,
>> 
>> I do not expect spawn can work with direct launch (e.g. srun)
>> 
>> Do you have PSM (e.g. Infinipath) hardware ? That could be linked to the 
>> failure
>> 
>> Can you please try
>> 
>> mpirun --mca pml ob1 --mca btl tcp,sm,self -np 1 --hostfile my_hosts 
>> ./manager 1
>> 
>> and see if it help ?
>> 
>> Note if you have the possibility, I suggest you first try that without 
>> slurm, and then within a slurm job
>> 
>> Cheers,
>> 
>> Gilles
>> 
>> On Thursday, September 29, 2016, juraj2...@gmail.com 
>> <javascript:_e(%7B%7D,'cvml','juraj2...@gmail.com');> <juraj2...@gmail.com 
>> <javascript:_e(%7B%7D,'cvml','juraj2...@gmail.com');>> wrote:
>> Hello,
>> 
>> I am using MPI_Comm_spawn to dynamically create new processes from single 
>> manager process. Everything works fine when all the processes are running on 
>> the same node. But imposing restriction to run only a single process per 
>> node does not work. Below are the errors produced during multinode 
>> interactive session and multinode sbatch job.
>> 
>> The system I am using is: Linux version 3.10.0-229.el7.x86_64 
>> (buil...@kbuilder.dev.centos.org <>) (gcc version 4.8.2 20140120 (Red Hat 
>> 4.8.2-16) (GCC) )
>> I am using Open MPI 2.0.1
>> Slurm is version 15.08.9
>> 
>> What is preventing my jobs to spawn on multiple nodes? Does slurm requires 
>> some additional configuration to allow it? Is it issue on the MPI side, does 
>> it need to be compiled with some special flag (I have compiled it with 
>> --enable-mpi-fortran=all --with-pmi)? 
>> 
>> The code I am launching is here: https://github.com/goghino/dynamicMPI 
>> <https://github.com/goghino/dynamicMPI>
>> 
>> Manager tries to launch one new process (./manager 1), the error produced by 
>> requesting each process to be located on different node (interactive 
>> session):
>> $ salloc -N 2
>> $ cat my_hosts
>> icsnode37
>> icsnode38
>> $ mpirun -np 1 -npernode 1 --hostfile my_hosts ./manager 1
>> [manager]I'm running MPI 3.1
>> [manager]Runing on node icsnode37
>> icsnode37.12614Assertion failure at ptl.c:183: epaddr == ((void *)0)
>> icsnode38.32443Assertion failure at ptl.c:183: epaddr == ((void *)0)
>> [icsnode37:12614] *** Process received signal ***
>> [icsnode37:12614] Signal: Aborted (6)
>> [icsnode37:12614] Signal code:  (-6)
>> [icsnode38:32443] *** Process received signal ***
>> [icsnode38:32443] Signal: Aborted (6)
>> [icsnode38:32443] Signal code:  (-6)
>> 
>> The same example as above via sbatch job submission:
>> $ cat job.sbatch
>> #!/bin/bash
>> 
>> #SBATCH --nodes=2
>> #SBATCH --ntasks-per-node=1
>> 
>> module load openmpi/2.0.1
>> srun -n 1 -N 1 ./manager 1
>> 
>> $ cat output.o
>> [manager]I'm running MPI 3.1
>> [manager]Runing on node icsnode39
>> srun: Job step aborted: Waiting up to 32 seconds for job step to finish.
>> [icsnode39:9692] *** An error occurred in MPI_Comm_spawn
>> [icsnode39:9692] *** reported by process [1007812608,0]
>> [icsnode39:9692] *** on communicator MPI_COMM_SELF
>> [icsnode39:9692] *** MPI_ERR_SPAWN: could not spawn processes
>> [icsnode39:9692] *** MPI_ERRORS_ARE_FATAL (processes in this communicator 
>> will now abort,
>> [icsnode39:9692] ***and potentially your MPI job)
>> In: PMI_Abort(50, N/A)
>> slurmstepd: *** STEP 15378.0 ON icsnode39 CANCELLED AT 2016-09-26T16:48:20 
>> ***
>> srun: error: icsnode39: task 0: Exited with exit code 50
>> 
>> Thank for any feedback!
>> 
>> Best regards,
>> Juraj
>> ___
>> users mailing list
>> users@lists.open-mpi.org 
>> <javascript:_e(%7B%7D,'cvml','users@lists.open-mpi.org');>
>> https://rfd.newmexicoconsortium.org/mailman/listinfo/users 
>> <https://rfd.newmexicoconsortium.org/mailman/listinfo/users>
> ___
> users mailing list
> users@lists.open-mpi.org
> https://rfd.newmexicoconsortium.org/mailman/listinfo/users

___
users mailing list
users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

Re: [OMPI users] MPI_Comm_spawn

[r...@open-mpi.org]

Spawn definitely does not work with srun. I don’t recognize the name of the 
file that segfaulted - what is “ptl.c”? Is that in your manager program?


> On Sep 29, 2016, at 6:06 AM, Gilles Gouaillardet 
>  wrote:
> 
> Hi,
> 
> I do not expect spawn can work with direct launch (e.g. srun)
> 
> Do you have PSM (e.g. Infinipath) hardware ? That could be linked to the 
> failure
> 
> Can you please try
> 
> mpirun --mca pml ob1 --mca btl tcp,sm,self -np 1 --hostfile my_hosts 
> ./manager 1
> 
> and see if it help ?
> 
> Note if you have the possibility, I suggest you first try that without slurm, 
> and then within a slurm job
> 
> Cheers,
> 
> Gilles
> 
> On Thursday, September 29, 2016, juraj2...@gmail.com 
>   > wrote:
> Hello,
> 
> I am using MPI_Comm_spawn to dynamically create new processes from single 
> manager process. Everything works fine when all the processes are running on 
> the same node. But imposing restriction to run only a single process per node 
> does not work. Below are the errors produced during multinode interactive 
> session and multinode sbatch job.
> 
> The system I am using is: Linux version 3.10.0-229.el7.x86_64 
> (buil...@kbuilder.dev.centos.org 
> ) (gcc 
> version 4.8.2 20140120 (Red Hat 4.8.2-16) (GCC) )
> I am using Open MPI 2.0.1
> Slurm is version 15.08.9
> 
> What is preventing my jobs to spawn on multiple nodes? Does slurm requires 
> some additional configuration to allow it? Is it issue on the MPI side, does 
> it need to be compiled with some special flag (I have compiled it with 
> --enable-mpi-fortran=all --with-pmi)? 
> 
> The code I am launching is here: https://github.com/goghino/dynamicMPI 
> 
> 
> Manager tries to launch one new process (./manager 1), the error produced by 
> requesting each process to be located on different node (interactive session):
> $ salloc -N 2
> $ cat my_hosts
> icsnode37
> icsnode38
> $ mpirun -np 1 -npernode 1 --hostfile my_hosts ./manager 1
> [manager]I'm running MPI 3.1
> [manager]Runing on node icsnode37
> icsnode37.12614Assertion failure at ptl.c:183: epaddr == ((void *)0)
> icsnode38.32443Assertion failure at ptl.c:183: epaddr == ((void *)0)
> [icsnode37:12614] *** Process received signal ***
> [icsnode37:12614] Signal: Aborted (6)
> [icsnode37:12614] Signal code:  (-6)
> [icsnode38:32443] *** Process received signal ***
> [icsnode38:32443] Signal: Aborted (6)
> [icsnode38:32443] Signal code:  (-6)
> 
> The same example as above via sbatch job submission:
> $ cat job.sbatch
> #!/bin/bash
> 
> #SBATCH --nodes=2
> #SBATCH --ntasks-per-node=1
> 
> module load openmpi/2.0.1
> srun -n 1 -N 1 ./manager 1
> 
> $ cat output.o
> [manager]I'm running MPI 3.1
> [manager]Runing on node icsnode39
> srun: Job step aborted: Waiting up to 32 seconds for job step to finish.
> [icsnode39:9692] *** An error occurred in MPI_Comm_spawn
> [icsnode39:9692] *** reported by process [1007812608,0]
> [icsnode39:9692] *** on communicator MPI_COMM_SELF
> [icsnode39:9692] *** MPI_ERR_SPAWN: could not spawn processes
> [icsnode39:9692] *** MPI_ERRORS_ARE_FATAL (processes in this communicator 
> will now abort,
> [icsnode39:9692] ***and potentially your MPI job)
> In: PMI_Abort(50, N/A)
> slurmstepd: *** STEP 15378.0 ON icsnode39 CANCELLED AT 2016-09-26T16:48:20 ***
> srun: error: icsnode39: task 0: Exited with exit code 50
> 
> Thank for any feedback!
> 
> Best regards,
> Juraj
> ___
> users mailing list
> users@lists.open-mpi.org
> https://rfd.newmexicoconsortium.org/mailman/listinfo/users

___
users mailing list
users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

Re: [OMPI users] Openmpi 1.10.x, mpirun and Slurm 15.08 problem

[r...@open-mpi.org]

This isn’t an issue with the SLURM integration - this is the problem of our OOB 
not correctly picking the right subnet for connecting back to mpirun. In this 
specific case, you probably want

-mca btl_tcp_if_include em4 -mca oob_tcp_if_include em4

since it is the em4 network that ties the compute nodes together, and the 
compute nodes to the frontend

We are working on the subnet selection logic, but the 1.10 series seems to have 
not been updated with those changes

> On Sep 23, 2016, at 6:00 AM, Marcin Krotkiewski 
>  wrote:
> 
> Hi,
> 
> I have stumbled upon a similar issue, so I wonder those might be related. On 
> one of our systems I get the following error message, both when using openmpi 
> 1.8.8 and 1.10.4
> 
> $ mpirun -debug-daemons --mca btl tcp,self --mca mca_base_verbose 100 --mca 
> btl_base_verbose 100 ls
> 
> [...]
> [compute-1-1.local:07302] mca: base: close: unloading component direct
> [compute-1-1.local:07302] mca: base: close: unloading component radix
> [compute-1-1.local:07302] mca: base: close: unloading component debruijn
> [compute-1-1.local:07302] orte_routed_base_select: initializing selected 
> component binomial
> [compute-1-2.local:13744] [[63041,0],2]: parent 0 num_children 0
> Daemon [[63041,0],2] checking in as pid 13744 on host c1-2
> [compute-1-2.local:13744] [[63041,0],2] orted: up and running - waiting for 
> commands!
> [compute-1-2.local:13744] [[63041,0],2] tcp_peer_send_blocking: send() to 
> socket 9 failed: Broken pipe (32)
> [compute-1-2.local:13744] mca: base: close: unloading component binomial
> [compute-1-1.local:07302] [[63041,0],1]: parent 0 num_children 0
> Daemon [[63041,0],1] checking in as pid 7302 on host c1-1
> [compute-1-1.local:07302] [[63041,0],1] orted: up and running - waiting for 
> commands!
> [compute-1-1.local:07302] [[63041,0],1] tcp_peer_send_blocking: send() to 
> socket 9 failed: Broken pipe (32)
> [compute-1-1.local:07302] mca: base: close: unloading component binomial
> srun: error: c1-1: task 0: Exited with exit code 1
> srun: Terminating job step 4538.1
> srun: Job step aborted: Waiting up to 32 seconds for job step to finish.
> srun: error: c1-2: task 1: Exited with exit code 1
> 
> 
> I have also tested version 2.0.1 - this one works without problems.
> 
> In my case the problem appears on one system with slurm versions 15.08.8 and 
> 15.08.12. On another system running 15.08.8 all is working fine, so I guess 
> it is not about SLURM version, but maybe system / network configuration?
> 
> Following that thought I have also noticed this thread:
> 
> http://users.open-mpi.narkive.com/PwJpWXLm/ompi-users-tcp-peer-send-blocking-send-to-socket-9-failed-broken-pipe-32-on-openvz-containers
>  
> 
> As Jeff suggested there, I tried to run with --mca btl_tcp_if_include em1 
> --mca oob_tcp_if_include em1, but got the same error.
> 
> Could these problems be related to interface naming / lack of infiniband? Or 
> to the fact that the front-end node, from which I execute mpirun, has a 
> different network configuration? The system, on which things don't work, only 
> has TCP  network interfaces:
> 
> em1, lo (frontend has em1, em4 - local compute network, lo)
> 
> while the cluster, on which openmpi does work, uses infiniband, and had the 
> following tcp interfaces:
> 
> eth0, eth1, ib0, lo
> 
> I would appreciate any hints..
> 
> Thanks!
> 
> Marcin
> 
> 
> On 04/01/2016 04:16 PM, Jeff Squyres (jsquyres) wrote:
>> Ralph --
>> 
>> What's the state of PMI integration with SLURM in the v1.10.x series?  (I 
>> haven't kept up with SLURM's recent releases to know if something broke 
>> between existing Open MPI releases and their new releases...?)
>> 
>> 
>> 
>>> On Mar 31, 2016, at 4:24 AM, Tommi T  wrote:
>>> 
>>> Hi,
>>> 
>>> stack:
>>> el6.7, mlnx ofed 3.1 (IB FDR) and slurm 15.08.9 (whithout *.la libs).
>>> 
>>> problem:
>>> OpenMPI 1.10.x built with pmi support does not work when trying to use 
>>> sbatch/salloc - mpirun combination. srun ompi_mpi_app works fine.
>>> 
>>> Older 1.8.x version works fine under same salloc session.
>>> 
>>> ./configure --with-slurm --with-verbs --with-hwloc=internal --with-pmi 
>>> --with-cuda=/appl/opt/cuda/7.5/ --with-pic --enable-shared 
>>> --enable-mpi-thread-multiple --enable-contrib-no-build=vt
>>> 
>>> 
>>> I tried 1.10.3a from git also.
>>> 
>>> 
>>> mpirun  -debug-daemons ./1103aompitest
>>> Daemon [[44437,0],1] checking in as pid 40979 on host g59
>>> Daemon [[44437,0],2] checking in as pid 23566 on host g60
>>> [g59:40979] [[44437,0],1] orted: up and running - waiting for commands!
>>> [g60:23566] [[44437,0],2] orted: up and running - waiting for commands!
>>> [g59:40979] [[44437,0],1] tcp_peer_send_blocking: send() to socket 9 
>>> failed: Broken pipe (32)
>>> [g59:40979] [[44437,0],1]:errmgr_default_orted.c(260) updating exit status 
>>> to 1
>>> [g60:23566] [[44437,0],2] tcp_peer_send_blocking: send() to socket 9 
>>> failed: Broken pipe 

Re: [OMPI users] Strange errors when running mpirun

[r...@open-mpi.org]

Try removing the “pmix” entries as well

> On Sep 22, 2016, at 2:19 AM, Justin Chang <jychan...@gmail.com> wrote:
> 
> "mpirun -n 1" was just to demonstrate that I get those error messages.
> I ran a simple helloworld.c and it still gives those two messages.
> 
> I did delete openmpi-sessions-* from my $TMPDIR but it doesn't solve
> the problem. Here's my $TMPDIR:
> 
> ~ cd $TMPDIR
> ~ pwd
> /var/folders/jd/qh5zn6jn5kz_byz9gxz5kl2mgn/T
> ~ ls
> MediaCache
> TemporaryItems
> com.apple.AddressBook.ContactsAccountsService
> com.apple.AddressBook.InternetAccountsBridge
> com.apple.AirPlayUIAgent
> com.apple.BKAgentService
> com.apple.CalendarAgent
> com.apple.CalendarAgent.CalNCService
> com.apple.CloudPhotosConfiguration
> com.apple.DataDetectorsDynamicData
> com.apple.ICPPhotoStreamLibraryService
> com.apple.InputMethodKit.TextReplacementService
> com.apple.PhotoIngestService
> com.apple.Preview
> com.apple.Safari
> com.apple.SocialPushAgent
> com.apple.WeatherKitService
> com.apple.cloudphotosd
> com.apple.dt.XCDocumenter.XCDocumenterExtension
> com.apple.dt.XcodeBuiltInExtensions
> com.apple.geod
> com.apple.iCal.CalendarNC
> com.apple.lateragent
> com.apple.ncplugin.stocks
> com.apple.ncplugin.weather
> com.apple.notificationcenterui.WeatherSummary
> com.apple.photolibraryd
> com.apple.photomoments
> com.apple.quicklook.ui.helper
> com.apple.soagent
> com.getdropbox.dropbox.garcon
> icdd501
> ics21406
> openmpi-sessions-501@Justins-MacBook-Pro-2_0
> pmix-12195
> pmix-12271
> pmix-12289
> pmix-12295
> pmix-12304
> pmix-12313
> pmix-12367
> pmix-12397
> pmix-12775
> pmix-12858
> pmix-17118
> pmix-1754
> pmix-20632
> pmix-20793
> pmix-20849
> pmix-21019
> pmix-22316
> pmix-8129
> pmix-8494
> xcrun_db
> ~ rm -rf openmpi-sessions-501@Justins-MacBook-Pro-2_0
> ~ mpirun -n 1
> [Justins-MacBook-Pro-2.local:22527] [[12992,0],0] bind() failed on
> error Address already in use (48)
> [Justins-MacBook-Pro-2.local:22527] [[12992,0],0] ORTE_ERROR_LOG:
> Error in file oob_usock_component.c at line 228
> --
> No executable was specified on the mpirun command line.
> 
> Aborting.
> --------------
> 
> and when I type "ls" the directory
> "openmpi-sessions-501@Justins-MacBook-Pro-2_0" reappeared. Unless
> there's a different directory I need to look for?
> 
> On Thu, Sep 22, 2016 at 4:08 AM, r...@open-mpi.org <r...@open-mpi.org> wrote:
>> Maybe I’m missing something, but “mpirun -n 1” doesn’t include the name of 
>> an application to execute.
>> 
>> The error message prior to that error indicates that you have some cruft 
>> sitting in your tmpdir. You just need to clean it out - look for something 
>> that starts with “openmpi”
>> 
>> 
>>> On Sep 22, 2016, at 1:45 AM, Justin Chang <jychan...@gmail.com> wrote:
>>> 
>>> Dear all,
>>> 
>>> So I upgraded/updated my Homebrew on my Macbook and installed Open MPI
>>> 2.0.1 using "brew install openmpi". However, when I open up a terminal
>>> and type "mpirun -n 1" I get the following messages:
>>> 
>>> ~ mpirun -n 1
>>> [Justins-MacBook-Pro-2.local:20793] [[13318,0],0] bind() failed on
>>> error Address already in use (48)
>>> [Justins-MacBook-Pro-2.local:20793] [[13318,0],0] ORTE_ERROR_LOG:
>>> Error in file oob_usock_component.c at line 228
>>> --
>>> No executable was specified on the mpirun command line.
>>> 
>>> Aborting.
>>> --
>>> 
>>> 
>>> I have never seen anything like the first two lines. I also installed
>>> python and mpi4py via pip, and when I still get the same messages:
>>> 
>>> ~ python -c "from mpi4py import MPI"
>>> [Justins-MacBook-Pro-2.local:20871] [[13496,0],0] bind() failed on
>>> error Address already in use (48)
>>> [Justins-MacBook-Pro-2.local:20871] [[13496,0],0] ORTE_ERROR_LOG:
>>> Error in file oob_usock_component.c at line 228
>>> 
>>> But now if I add "mpirun -n 1" I get the following:
>>> 
>>> ~ mpirun -n 1 python -c "from mpi4py import MPI"
>>> [Justins-MacBook-Pro-2.local:20935] [[13560,0],0] bind() failed on
>>> error Address already in use (48)
>>> [Justins-MacBook-

Re: [OMPI users] Strange errors when running mpirun

[r...@open-mpi.org]

Maybe I’m missing something, but “mpirun -n 1” doesn’t include the name of an 
application to execute.

The error message prior to that error indicates that you have some cruft 
sitting in your tmpdir. You just need to clean it out - look for something that 
starts with “openmpi”


> On Sep 22, 2016, at 1:45 AM, Justin Chang  wrote:
> 
> Dear all,
> 
> So I upgraded/updated my Homebrew on my Macbook and installed Open MPI
> 2.0.1 using "brew install openmpi". However, when I open up a terminal
> and type "mpirun -n 1" I get the following messages:
> 
> ~ mpirun -n 1
> [Justins-MacBook-Pro-2.local:20793] [[13318,0],0] bind() failed on
> error Address already in use (48)
> [Justins-MacBook-Pro-2.local:20793] [[13318,0],0] ORTE_ERROR_LOG:
> Error in file oob_usock_component.c at line 228
> --
> No executable was specified on the mpirun command line.
> 
> Aborting.
> --
> 
> 
> I have never seen anything like the first two lines. I also installed
> python and mpi4py via pip, and when I still get the same messages:
> 
> ~ python -c "from mpi4py import MPI"
> [Justins-MacBook-Pro-2.local:20871] [[13496,0],0] bind() failed on
> error Address already in use (48)
> [Justins-MacBook-Pro-2.local:20871] [[13496,0],0] ORTE_ERROR_LOG:
> Error in file oob_usock_component.c at line 228
> 
> But now if I add "mpirun -n 1" I get the following:
> 
> ~ mpirun -n 1 python -c "from mpi4py import MPI"
> [Justins-MacBook-Pro-2.local:20935] [[13560,0],0] bind() failed on
> error Address already in use (48)
> [Justins-MacBook-Pro-2.local:20935] [[13560,0],0] ORTE_ERROR_LOG:
> Error in file oob_usock_component.c at line 228
> [Justins-MacBook-Pro-2.local:20936] [[13560,1],0]
> usock_peer_send_blocking: send() to socket 17 failed: Socket is not
> connected (57)
> [Justins-MacBook-Pro-2.local:20936] [[13560,1],0] ORTE_ERROR_LOG:
> Unreachable in file oob_usock_connection.c at line 315
> [Justins-MacBook-Pro-2.local:20936] [[13560,1],0]
> orte_usock_peer_try_connect: usock_peer_send_connect_ack to proc
> [[13560,0],0] failed: Unreachable (-12)
> [Justins-MacBook-Pro-2:20936] *** Process received signal ***
> [Justins-MacBook-Pro-2:20936] Signal: Segmentation fault: 11 (11)
> [Justins-MacBook-Pro-2:20936] Signal code:  (0)
> [Justins-MacBook-Pro-2:20936] Failing at address: 0x0
> ---
> Primary job  terminated normally, but 1 process returned
> a non-zero exit code.. Per user-direction, the job has been aborted.
> ---
> --
> mpirun detected that one or more processes exited with non-zero
> status, thus causing
> the job to be terminated. The first process to do so was:
> 
>  Process name: [[13560,1],0]
>  Exit code:1
> --
> 
> Clearly something is wrong here. I already tried things like "rm -rf
> $TMPDIR/openmpi-sessions-*" but said directory keeps reappearing and
> the error persists. Why does this happen and how do I fix it? For what
> it's worth, here's some other information that may help:
> 
> ~ mpicc --version
> Apple LLVM version 8.0.0 (clang-800.0.38)
> Target: x86_64-apple-darwin15.6.0
> Thread model: posix
> InstalledDir: 
> /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin
> 
> I tested Hello World with both mpicc and mpif90, and they still work
> despite showing those two error/warning messages.
> 
> Thanks,
> Justin
> ___
> users mailing list
> users@lists.open-mpi.org
> https://rfd.newmexicoconsortium.org/mailman/listinfo/users

___
users mailing list
users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

Re: [OMPI users] OMPI users] Unable to mpirun from within torque

[r...@open-mpi.org]

I’m pruning this email thread so I can actually read the blasted thing :-)

Guys: you are off in the wilderness chasing ghosts! Please stop.

When I say that Torque uses an “ordered” file, I am _not_ saying that all the 
host entries of the same name have to be listed consecutively. I am saying that 
the _position_ of each entry has meaning, and you cannot just change it.

I have honestly totally lost the root of this discussion in all the white noise 
about the PBS_NODEFILE. Can we reboot?
Ralph


> On Sep 8, 2016, at 5:26 AM, Gilles Gouaillardet 
>  wrote:
> 
> Oswin,
> 
> One more thing, can you
> 
> pbsdsh -v hostname
> 
> before invoking mpirun ?
> Hopefully this should print the three hostnames
> 
> Then you can
> ldd `which pbsdsh`
> And see which libtorque.so is linked with it
> 
> Cheers,
> 
> Gilles
> 
> 

___
users mailing list
users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

Re: [OMPI users] Unable to mpirun from within torque

[r...@open-mpi.org]

If you are correctly analyzing things, then there would be an issue in the 
code. When we get an allocation from a resource manager, we set a flag 
indicating that it is “gospel” - i.e., that we do not directly sense the number 
of cores on a node and set the #slots equal to that value. Instead, we take the 
RM-provided allocation as ultimate truth.

This should be true even if you add a machinefile, as the machinefile is only 
used to “filter” the nodelist provided by the RM. It shouldn’t cause the #slots 
to be modified.

Taking a quick glance at the v2.x code, it looks to me like all is being done 
correctly. Again, output from a debug build would resolve that question


> On Sep 7, 2016, at 10:56 PM, Gilles Gouaillardet  wrote:
> 
> Oswin,
> 
> 
> unfortunatly some important info is missing.
> 
> i guess the root cause is Open MPI was not configure'd with --enable-debug
> 
> 
> could you please update your torque script and simply add the following 
> snippet before invoking mpirun
> 
> 
> echo PBS_NODEFILE
> 
> cat $PBS_NODEFILE
> 
> echo ---
> 
> 
> as i wrote in an other email, i suspect hosts are not ordered (and i'd like 
> to confirm that) and Open MPI does not handle that correctly
> 
> 
> Cheers,
> 
> 
> Gilles
> 
> On 9/7/2016 10:25 PM, Oswin Krause wrote:
>> Hi Gilles,
>> 
>> Thanks for the hint with the machinefile. I know it is not equivalent and i 
>> do not intend to use that approach. I just wanted to know whether I could 
>> start the program successfully at all.
>> 
>> Outside torque(4.2), rsh seems to be used which works fine, querying a 
>> password if no kerberos ticket is there
>> 
>> Here is the output:
>> [zbh251@a00551 ~]$ mpirun -V
>> mpirun (Open MPI) 2.0.1
>> [zbh251@a00551 ~]$ ompi_info | grep ras
>> MCA ras: loadleveler (MCA v2.1.0, API v2.0.0, Component 
>> v2.0.1)
>> MCA ras: simulator (MCA v2.1.0, API v2.0.0, Component v2.0.1)
>> MCA ras: slurm (MCA v2.1.0, API v2.0.0, Component v2.0.1)
>> MCA ras: tm (MCA v2.1.0, API v2.0.0, Component v2.0.1)
>> [zbh251@a00551 ~]$ mpirun --mca plm_base_verbose 10 --tag-output 
>> -display-map hostname
>> [a00551.science.domain:04104] mca: base: components_register: registering 
>> framework plm components
>> [a00551.science.domain:04104] mca: base: components_register: found loaded 
>> component isolated
>> [a00551.science.domain:04104] mca: base: components_register: component 
>> isolated has no register or open function
>> [a00551.science.domain:04104] mca: base: components_register: found loaded 
>> component rsh
>> [a00551.science.domain:04104] mca: base: components_register: component rsh 
>> register function successful
>> [a00551.science.domain:04104] mca: base: components_register: found loaded 
>> component slurm
>> [a00551.science.domain:04104] mca: base: components_register: component 
>> slurm register function successful
>> [a00551.science.domain:04104] mca: base: components_register: found loaded 
>> component tm
>> [a00551.science.domain:04104] mca: base: components_register: component tm 
>> register function successful
>> [a00551.science.domain:04104] mca: base: components_open: opening plm 
>> components
>> [a00551.science.domain:04104] mca: base: components_open: found loaded 
>> component isolated
>> [a00551.science.domain:04104] mca: base: components_open: component isolated 
>> open function successful
>> [a00551.science.domain:04104] mca: base: components_open: found loaded 
>> component rsh
>> [a00551.science.domain:04104] mca: base: components_open: component rsh open 
>> function successful
>> [a00551.science.domain:04104] mca: base: components_open: found loaded 
>> component slurm
>> [a00551.science.domain:04104] mca: base: components_open: component slurm 
>> open function successful
>> [a00551.science.domain:04104] mca: base: components_open: found loaded 
>> component tm
>> [a00551.science.domain:04104] mca: base: components_open: component tm open 
>> function successful
>> [a00551.science.domain:04104] mca:base:select: Auto-selecting plm components
>> [a00551.science.domain:04104] mca:base:select:(  plm) Querying component 
>> [isolated]
>> [a00551.science.domain:04104] mca:base:select:(  plm) Query of component 
>> [isolated] set priority to 0
>> [a00551.science.domain:04104] mca:base:select:(  plm) Querying component 
>> [rsh]
>> [a00551.science.domain:04104] mca:base:select:(  plm) Query of component 
>> [rsh] set priority to 10
>> [a00551.science.domain:04104] mca:base:select:(  plm) Querying component 
>> [slurm]
>> [a00551.science.domain:04104] mca:base:select:(  plm) Querying component [tm]
>> [a00551.science.domain:04104] mca:base:select:(  plm) Query of component 
>> [tm] set priority to 75
>> [a00551.science.domain:04104] mca:base:select:(  plm) Selected component [tm]
>> [a00551.science.domain:04104] mca: base: close: component isolated closed
>> [a00551.science.domain:04104] mca: base: close: unloading 

Re: [OMPI users] OMPI users] Unable to mpirun from within torque

[r...@open-mpi.org]

Someone has done some work there since I last did, but I can see the issue. 
Torque indeed always provides an ordered file - the only way you can get an 
unordered one is for someone to edit it, and that is forbidden - i.e., you get 
what you deserve because you are messing around with a system-defined file :-)

The problem is that Torque internally assigns a “launch ID” which is just the 
integer position of the nodename in the PBS_NODEFILE. So if you modify that 
position, then we get the wrong index - and everything goes down the drain from 
there. In your example, n1.cluster changed index from 3 to 2 because of your 
edit. Torque thinks that index 2 is just another reference to n0.cluster, and 
so we merrily launch a daemon onto the wrong node.

They have a good reason for doing things this way. It allows you to launch a 
process against each launch ID, and the pattern will reflect the original qsub 
request in what we would call a map-by slot round-robin mode. This maximizes 
the use of shared memory, and is expected to provide good performance for a 
range of apps.

Lesson to be learned: never, ever muddle around with a system-generated file. 
If you want to modify where things go, then use one or more of the mpirun 
options to do so. We give you lots and lots of knobs for just that reason.



> On Sep 7, 2016, at 10:53 PM, Gilles Gouaillardet <gil...@rist.or.jp> wrote:
> 
> Ralph,
> 
> 
> there might be an issue within Open MPI.
> 
> 
> on the cluster i used, hostname returns the FQDN, and $PBS_NODEFILE uses the 
> FQDN too.
> 
> my $PBS_NODEFILE has one line per task, and it is ordered
> 
> e.g.
> 
> n0.cluster
> 
> n0.cluster
> 
> n1.cluster
> 
> n1.cluster
> 
> 
> in my torque script, i rewrote the machinefile like this
> 
> n0.cluster
> 
> n1.cluster
> 
> n0.cluster
> 
> n1.cluster
> 
> and updated the PBS environment variable to point to my new file.
> 
> 
> then i invoked
> 
> mpirun hostname
> 
> 
> 
> in the first case, 2 tasks run on n0 and 2 tasks run on n1
> in the second case, 4 tasks run on n0, and none on n1.
> 
> so i am thinking we might not support unordered $PBS_NODEFILE.
> 
> as a reminder, the submit command was
> qsub -l nodes=3:ppn=1
> but for some reasons i ignore, only two nodes were allocated (two slots on 
> the first one, one on the second one)
> and if i understand correctly, $PBS_NODEFILE was not ordered.
> (e.g. n0 n1 n0 and *not * n0 n0 n1)
> 
> i tried to reproduce this without hacking $PBS_NODEFILE, but my jobs hang in 
> the queue if only two nodes with 16 slots each are available and i request
> -l nodes=3:ppn=1
> i guess this is a different scheduler configuration, and i cannot change that.
> 
> Could you please have a look at this ?
> 
> Cheers,
> 
> Gilles
> 
> On 9/7/2016 11:15 PM, r...@open-mpi.org wrote:
>> The usual cause of this problem is that the nodename in the machinefile is 
>> given as a00551, while Torque is assigning the node name as 
>> a00551.science.domain. Thus, mpirun thinks those are two separate nodes and 
>> winds up spawning an orted on its own node.
>> 
>> You might try ensuring that your machinefile is using the exact same name as 
>> provided in your allocation
>> 
>> 
>>> On Sep 7, 2016, at 7:06 AM, Gilles Gouaillardet 
>>> <gilles.gouaillar...@gmail.com> wrote:
>>> 
>>> Thanjs for the ligs
>>> 
>>> From what i see now, it looks like a00551 is running both mpirun and orted, 
>>> though it should only run mpirun, and orted should run only on a00553
>>> 
>>> I will check the code and see what could be happening here
>>> 
>>> Btw, what is the output of
>>> hostname
>>> hostname -f
>>> On a00551 ?
>>> 
>>> Out of curiosity, is a previous version of Open MPI (e.g. v1.10.4) 
>>> installled and running correctly on your cluster ?
>>> 
>>> Cheers,
>>> 
>>> Gilles
>>> 
>>> Oswin Krause <oswin.kra...@ruhr-uni-bochum.de> wrote:
>>>> Hi Gilles,
>>>> 
>>>> Thanks for the hint with the machinefile. I know it is not equivalent
>>>> and i do not intend to use that approach. I just wanted to know whether
>>>> I could start the program successfully at all.
>>>> 
>>>> Outside torque(4.2), rsh seems to be used which works fine, querying a
>>>> password if no kerberos ticket is there
>>>> 
>>>> Here is the output:
>>>> [zbh251@a00551 ~]$ mpirun -V
>>>> mpirun (Open MPI) 2.0.1
>>>> [zbh251@a00551 ~]$ ompi_info | grep ras
>>>

Re: [OMPI users] OMPI users] Unable to mpirun from within torque

[r...@open-mpi.org]

You aren’t looking in the right place - there is an “openmpi” directory 
underneath that one, and the mca_xxx libraries are down there

> On Sep 7, 2016, at 7:43 AM, Oswin Krause  
> wrote:
> 
> Hi Gilles,
> 
> I do not have this library. Maybe this helps already...
> 
> libmca_common_sm.so  libmpi_mpifh.so  libmpi_usempif08.so  
> libompitrace.so  libopen-rte.so
> libmpi_cxx.solibmpi.solibmpi_usempi_ignore_tkr.so  
> libopen-pal.so   liboshmem.so
> 
> and mpirun does only link to libopen-pal/libopen-rte (aside the standard 
> stuff)
> 
> But still it is telling me that it has support for tm? libtorque is there and 
> the headers are also there and since i have enabled tm...*sigh*
> 
> Thanks again!
> 
> Oswin
> 
> On 2016-09-07 16:21, Gilles Gouaillardet wrote:
>> Note the torque library will only show up if you configure'd with
>> --disable-dlopen. Otherwise, you can ldd
>> /.../lib/openmpi/mca_plm_tm.so
>> Cheers,
>> Gilles
>> Bennet Fauber  wrote:
>>> Oswin,
>>> Does the torque library show up if you run
>>> $ ldd mpirun
>>> That would indicate that Torque support is compiled in.
>>> Also, what happens if you use the same hostfile, or some hostfile as
>>> an explicit argument when you run mpirun from within the torque job?
>>> -- bennet
>>> On Wed, Sep 7, 2016 at 9:25 AM, Oswin Krause
>>>  wrote:
 Hi Gilles,
 Thanks for the hint with the machinefile. I know it is not equivalent and i
 do not intend to use that approach. I just wanted to know whether I could
 start the program successfully at all.
 Outside torque(4.2), rsh seems to be used which works fine, querying a
 password if no kerberos ticket is there
 Here is the output:
 [zbh251@a00551 ~]$ mpirun -V
 mpirun (Open MPI) 2.0.1
 [zbh251@a00551 ~]$ ompi_info | grep ras
 MCA ras: loadleveler (MCA v2.1.0, API v2.0.0, Component
 v2.0.1)
 MCA ras: simulator (MCA v2.1.0, API v2.0.0, Component
 v2.0.1)
 MCA ras: slurm (MCA v2.1.0, API v2.0.0, Component v2.0.1)
 MCA ras: tm (MCA v2.1.0, API v2.0.0, Component v2.0.1)
 [zbh251@a00551 ~]$ mpirun --mca plm_base_verbose 10 --tag-output
 -display-map hostname
 [a00551.science.domain:04104] mca: base: components_register: registering
 framework plm components
 [a00551.science.domain:04104] mca: base: components_register: found loaded
 component isolated
 [a00551.science.domain:04104] mca: base: components_register: component
 isolated has no register or open function
 [a00551.science.domain:04104] mca: base: components_register: found loaded
 component rsh
 [a00551.science.domain:04104] mca: base: components_register: component rsh
 register function successful
 [a00551.science.domain:04104] mca: base: components_register: found loaded
 component slurm
 [a00551.science.domain:04104] mca: base: components_register: component
 slurm register function successful
 [a00551.science.domain:04104] mca: base: components_register: found loaded
 component tm
 [a00551.science.domain:04104] mca: base: components_register: component tm
 register function successful
 [a00551.science.domain:04104] mca: base: components_open: opening plm
 components
 [a00551.science.domain:04104] mca: base: components_open: found loaded
 component isolated
 [a00551.science.domain:04104] mca: base: components_open: component 
 isolated
 open function successful
 [a00551.science.domain:04104] mca: base: components_open: found loaded
 component rsh
 [a00551.science.domain:04104] mca: base: components_open: component rsh 
 open
 function successful
 [a00551.science.domain:04104] mca: base: components_open: found loaded
 component slurm
 [a00551.science.domain:04104] mca: base: components_open: component slurm
 open function successful
 [a00551.science.domain:04104] mca: base: components_open: found loaded
 component tm
 [a00551.science.domain:04104] mca: base: components_open: component tm open
 function successful
 [a00551.science.domain:04104] mca:base:select: Auto-selecting plm 
 components
 [a00551.science.domain:04104] mca:base:select:(  plm) Querying component
 [isolated]
 [a00551.science.domain:04104] mca:base:select:(  plm) Query of component
 [isolated] set priority to 0
 [a00551.science.domain:04104] mca:base:select:(  plm) Querying component
 [rsh]
 [a00551.science.domain:04104] mca:base:select:(  plm) Query of component
 [rsh] set priority to 10
 [a00551.science.domain:04104] mca:base:select:(  plm) Querying component
 [slurm]
 [a00551.science.domain:04104] mca:base:select:(  plm) Querying component
 [tm]
 [a00551.science.domain:04104] mca:base:select:(  plm) Query of component
 [tm] 

Re: [OMPI users] OMPI users] Unable to mpirun from within torque

[r...@open-mpi.org]

The usual cause of this problem is that the nodename in the machinefile is 
given as a00551, while Torque is assigning the node name as 
a00551.science.domain. Thus, mpirun thinks those are two separate nodes and 
winds up spawning an orted on its own node.

You might try ensuring that your machinefile is using the exact same name as 
provided in your allocation


> On Sep 7, 2016, at 7:06 AM, Gilles Gouaillardet 
>  wrote:
> 
> Thanjs for the ligs
> 
> From what i see now, it looks like a00551 is running both mpirun and orted, 
> though it should only run mpirun, and orted should run only on a00553
> 
> I will check the code and see what could be happening here
> 
> Btw, what is the output of
> hostname
> hostname -f
> On a00551 ?
> 
> Out of curiosity, is a previous version of Open MPI (e.g. v1.10.4) installled 
> and running correctly on your cluster ?
> 
> Cheers,
> 
> Gilles
> 
> Oswin Krause  wrote:
>> Hi Gilles,
>> 
>> Thanks for the hint with the machinefile. I know it is not equivalent 
>> and i do not intend to use that approach. I just wanted to know whether 
>> I could start the program successfully at all.
>> 
>> Outside torque(4.2), rsh seems to be used which works fine, querying a 
>> password if no kerberos ticket is there
>> 
>> Here is the output:
>> [zbh251@a00551 ~]$ mpirun -V
>> mpirun (Open MPI) 2.0.1
>> [zbh251@a00551 ~]$ ompi_info | grep ras
>> MCA ras: loadleveler (MCA v2.1.0, API v2.0.0, Component 
>> v2.0.1)
>> MCA ras: simulator (MCA v2.1.0, API v2.0.0, Component 
>> v2.0.1)
>> MCA ras: slurm (MCA v2.1.0, API v2.0.0, Component 
>> v2.0.1)
>> MCA ras: tm (MCA v2.1.0, API v2.0.0, Component v2.0.1)
>> [zbh251@a00551 ~]$ mpirun --mca plm_base_verbose 10 --tag-output 
>> -display-map hostname
>> [a00551.science.domain:04104] mca: base: components_register: 
>> registering framework plm components
>> [a00551.science.domain:04104] mca: base: components_register: found 
>> loaded component isolated
>> [a00551.science.domain:04104] mca: base: components_register: component 
>> isolated has no register or open function
>> [a00551.science.domain:04104] mca: base: components_register: found 
>> loaded component rsh
>> [a00551.science.domain:04104] mca: base: components_register: component 
>> rsh register function successful
>> [a00551.science.domain:04104] mca: base: components_register: found 
>> loaded component slurm
>> [a00551.science.domain:04104] mca: base: components_register: component 
>> slurm register function successful
>> [a00551.science.domain:04104] mca: base: components_register: found 
>> loaded component tm
>> [a00551.science.domain:04104] mca: base: components_register: component 
>> tm register function successful
>> [a00551.science.domain:04104] mca: base: components_open: opening plm 
>> components
>> [a00551.science.domain:04104] mca: base: components_open: found loaded 
>> component isolated
>> [a00551.science.domain:04104] mca: base: components_open: component 
>> isolated open function successful
>> [a00551.science.domain:04104] mca: base: components_open: found loaded 
>> component rsh
>> [a00551.science.domain:04104] mca: base: components_open: component rsh 
>> open function successful
>> [a00551.science.domain:04104] mca: base: components_open: found loaded 
>> component slurm
>> [a00551.science.domain:04104] mca: base: components_open: component 
>> slurm open function successful
>> [a00551.science.domain:04104] mca: base: components_open: found loaded 
>> component tm
>> [a00551.science.domain:04104] mca: base: components_open: component tm 
>> open function successful
>> [a00551.science.domain:04104] mca:base:select: Auto-selecting plm 
>> components
>> [a00551.science.domain:04104] mca:base:select:(  plm) Querying component 
>> [isolated]
>> [a00551.science.domain:04104] mca:base:select:(  plm) Query of component 
>> [isolated] set priority to 0
>> [a00551.science.domain:04104] mca:base:select:(  plm) Querying component 
>> [rsh]
>> [a00551.science.domain:04104] mca:base:select:(  plm) Query of component 
>> [rsh] set priority to 10
>> [a00551.science.domain:04104] mca:base:select:(  plm) Querying component 
>> [slurm]
>> [a00551.science.domain:04104] mca:base:select:(  plm) Querying component 
>> [tm]
>> [a00551.science.domain:04104] mca:base:select:(  plm) Query of component 
>> [tm] set priority to 75
>> [a00551.science.domain:04104] mca:base:select:(  plm) Selected component 
>> [tm]
>> [a00551.science.domain:04104] mca: base: close: component isolated 
>> closed
>> [a00551.science.domain:04104] mca: base: close: unloading component 
>> isolated
>> [a00551.science.domain:04104] mca: base: close: component rsh closed
>> [a00551.science.domain:04104] mca: base: close: unloading component rsh
>> [a00551.science.domain:04104] mca: base: close: component slurm closed
>> [a00551.science.domain:04104] mca: base: close: unloading 

Re: [OMPI users] stdin issue with openmpi/2.0.0

[r...@open-mpi.org]

Oh my - that indeed illustrated the problem!! It is indeed a race condition on 
the backend orted. I’ll try to fix it - probably have to send you a patch to 
test?

> On Aug 30, 2016, at 1:04 PM, Jingchao Zhang <zh...@unl.edu> wrote:
> 
> $mpirun -mca state_base_verbose 5 ./a.out < test.in
> 
> Please see attached for the outputs.
> 
> Thank you Ralph. I am willing to provide whatever information you need.
> 
> From: users <users-boun...@lists.open-mpi.org 
> <mailto:users-boun...@lists.open-mpi.org>> on behalf of r...@open-mpi.org 
> <mailto:r...@open-mpi.org> <r...@open-mpi.org <mailto:r...@open-mpi.org>>
> Sent: Tuesday, August 30, 2016 1:45:45 PM
> To: Open MPI Users
> Subject: Re: [OMPI users] stdin issue with openmpi/2.0.0
>  
> Well, that helped a bit. For some reason, your system is skipping a step in 
> the launch state machine, and so we never hit the step where we setup the IO 
> forwarding system.
> 
> Sorry to keep poking, but I haven’t seen this behavior anywhere else, and so 
> I have no way to replicate it. Must be a subtle race condition.
> 
> Can you replace “plm” with ‘“state” and try to hit a “bad” run again?
> 
> 
>> On Aug 30, 2016, at 12:30 PM, Jingchao Zhang <zh...@unl.edu 
>> <mailto:zh...@unl.edu>> wrote:
>> 
>> Yes, all procs were launched properly. I added “-mca plm_base_verbose 5” to 
>> the mpirun command. Please see attached for the results.
>> 
>> $mpirun -mca plm_base_verbose 5 ./a.out < test.in
>> 
>> I mentioned in my initial post that the test job can run properly for the 
>> 1st time. But if I kill the job and resubmit, then it hangs. It happened 
>> with the job above as well. Very odd. 
>> From: users <users-boun...@lists.open-mpi.org 
>> <mailto:users-boun...@lists.open-mpi.org>> on behalf of r...@open-mpi.org 
>> <mailto:r...@open-mpi.org> <r...@open-mpi.org <mailto:r...@open-mpi.org>>
>> Sent: Tuesday, August 30, 2016 12:56:33 PM
>> To: Open MPI Users
>> Subject: Re: [OMPI users] stdin issue with openmpi/2.0.0
>>  
>> Hmmm...well, the problem appears to be that we aren’t setting up the input 
>> channel to read stdin. This happens immediately after the application is 
>> launched - there is no “if” clause or anything else in front of it. The only 
>> way it wouldn’t get called is if all the procs weren’t launched, but that 
>> appears to be happening, yes?
>> 
>> Hence my confusion - there is no test in front of that print statement now, 
>> and yet we aren’t seeing the code being called.
>> 
>> Could you please add “-mca plm_base_verbose 5” to your cmd line? We should 
>> see a debug statement print that contains "plm:base:launch wiring up iof for 
>> job”
>> 
>> 
>> 
>>> On Aug 30, 2016, at 11:40 AM, Jingchao Zhang <zh...@unl.edu 
>>> <mailto:zh...@unl.edu>> wrote:
>>> 
>>> I checked again and as far as I can tell, everything was setup correctly. I 
>>> added "HCC debug" to the output message to make sure it's the correct 
>>> plugin. 
>>> 
>>> The updated outputs:
>>> $ mpirun ./a.out < test.in
>>> [c1725.crane.hcc.unl.edu <http://c1725.crane.hcc.unl.edu/>:218844] HCC 
>>> debug: [[26513,0],0] iof:hnp pushing fd 35 for process [[26513,1],0]
>>> [c1725.crane.hcc.unl.edu <http://c1725.crane.hcc.unl.edu/>:218844] HCC 
>>> debug: [[26513,0],0] iof:hnp pushing fd 41 for process [[26513,1],0]
>>> [c1725.crane.hcc.unl.edu <http://c1725.crane.hcc.unl.edu/>:218844] HCC 
>>> debug: [[26513,0],0] iof:hnp pushing fd 43 for process [[26513,1],0]
>>> [c1725.crane.hcc.unl.edu <http://c1725.crane.hcc.unl.edu/>:218844] HCC 
>>> debug: [[26513,0],0] iof:hnp pushing fd 37 for process [[26513,1],1]
>>> [c1725.crane.hcc.unl.edu <http://c1725.crane.hcc.unl.edu/>:218844] HCC 
>>> debug: [[26513,0],0] iof:hnp pushing fd 46 for process [[26513,1],1]
>>> [c1725.crane.hcc.unl.edu <http://c1725.crane.hcc.unl.edu/>:218844] HCC 
>>> debug: [[26513,0],0] iof:hnp pushing fd 49 for process [[26513,1],1]
>>> [c1725.crane.hcc.unl.edu <http://c1725.crane.hcc.unl.edu/>:218844] HCC 
>>> debug: [[26513,0],0] iof:hnp pushing fd 38 for process [[26513,1],2]
>>> [c1725.crane.hcc.unl.edu <http://c1725.crane.hcc.unl.edu/>:218844] HCC 
>>> debug: [[26513,0],0] iof:hnp pushing fd 50 for process [[26513,1],2]
>>> [c1725.crane.hcc.unl.edu <http://c1725.crane.hcc.unl.edu/>:218844] HCC 
>>> debug: [[26513,0],0] iof:hnp pushing fd 5

Re: [OMPI users] stdin issue with openmpi/2.0.0

[r...@open-mpi.org]

Well, that helped a bit. For some reason, your system is skipping a step in the 
launch state machine, and so we never hit the step where we setup the IO 
forwarding system.

Sorry to keep poking, but I haven’t seen this behavior anywhere else, and so I 
have no way to replicate it. Must be a subtle race condition.

Can you replace “plm” with ‘“state” and try to hit a “bad” run again?


> On Aug 30, 2016, at 12:30 PM, Jingchao Zhang <zh...@unl.edu> wrote:
> 
> Yes, all procs were launched properly. I added “-mca plm_base_verbose 5” to 
> the mpirun command. Please see attached for the results.
> 
> $mpirun -mca plm_base_verbose 5 ./a.out < test.in
> 
> I mentioned in my initial post that the test job can run properly for the 1st 
> time. But if I kill the job and resubmit, then it hangs. It happened with the 
> job above as well. Very odd. 
> From: users <users-boun...@lists.open-mpi.org 
> <mailto:users-boun...@lists.open-mpi.org>> on behalf of r...@open-mpi.org 
> <mailto:r...@open-mpi.org> <r...@open-mpi.org <mailto:r...@open-mpi.org>>
> Sent: Tuesday, August 30, 2016 12:56:33 PM
> To: Open MPI Users
> Subject: Re: [OMPI users] stdin issue with openmpi/2.0.0
>  
> Hmmm...well, the problem appears to be that we aren’t setting up the input 
> channel to read stdin. This happens immediately after the application is 
> launched - there is no “if” clause or anything else in front of it. The only 
> way it wouldn’t get called is if all the procs weren’t launched, but that 
> appears to be happening, yes?
> 
> Hence my confusion - there is no test in front of that print statement now, 
> and yet we aren’t seeing the code being called.
> 
> Could you please add “-mca plm_base_verbose 5” to your cmd line? We should 
> see a debug statement print that contains "plm:base:launch wiring up iof for 
> job”
> 
> 
> 
>> On Aug 30, 2016, at 11:40 AM, Jingchao Zhang <zh...@unl.edu 
>> <mailto:zh...@unl.edu>> wrote:
>> 
>> I checked again and as far as I can tell, everything was setup correctly. I 
>> added "HCC debug" to the output message to make sure it's the correct 
>> plugin. 
>> 
>> The updated outputs:
>> $ mpirun ./a.out < test.in
>> [c1725.crane.hcc.unl.edu <http://c1725.crane.hcc.unl.edu/>:218844] HCC 
>> debug: [[26513,0],0] iof:hnp pushing fd 35 for process [[26513,1],0]
>> [c1725.crane.hcc.unl.edu <http://c1725.crane.hcc.unl.edu/>:218844] HCC 
>> debug: [[26513,0],0] iof:hnp pushing fd 41 for process [[26513,1],0]
>> [c1725.crane.hcc.unl.edu <http://c1725.crane.hcc.unl.edu/>:218844] HCC 
>> debug: [[26513,0],0] iof:hnp pushing fd 43 for process [[26513,1],0]
>> [c1725.crane.hcc.unl.edu <http://c1725.crane.hcc.unl.edu/>:218844] HCC 
>> debug: [[26513,0],0] iof:hnp pushing fd 37 for process [[26513,1],1]
>> [c1725.crane.hcc.unl.edu <http://c1725.crane.hcc.unl.edu/>:218844] HCC 
>> debug: [[26513,0],0] iof:hnp pushing fd 46 for process [[26513,1],1]
>> [c1725.crane.hcc.unl.edu <http://c1725.crane.hcc.unl.edu/>:218844] HCC 
>> debug: [[26513,0],0] iof:hnp pushing fd 49 for process [[26513,1],1]
>> [c1725.crane.hcc.unl.edu <http://c1725.crane.hcc.unl.edu/>:218844] HCC 
>> debug: [[26513,0],0] iof:hnp pushing fd 38 for process [[26513,1],2]
>> [c1725.crane.hcc.unl.edu <http://c1725.crane.hcc.unl.edu/>:218844] HCC 
>> debug: [[26513,0],0] iof:hnp pushing fd 50 for process [[26513,1],2]
>> [c1725.crane.hcc.unl.edu <http://c1725.crane.hcc.unl.edu/>:218844] HCC 
>> debug: [[26513,0],0] iof:hnp pushing fd 52 for process [[26513,1],2]
>> [c1725.crane.hcc.unl.edu <http://c1725.crane.hcc.unl.edu/>:218844] HCC 
>> debug: [[26513,0],0] iof:hnp pushing fd 42 for process [[26513,1],3]
>> [c1725.crane.hcc.unl.edu <http://c1725.crane.hcc.unl.edu/>:218844] HCC 
>> debug: [[26513,0],0] iof:hnp pushing fd 53 for process [[26513,1],3]
>> [c1725.crane.hcc.unl.edu <http://c1725.crane.hcc.unl.edu/>:218844] HCC 
>> debug: [[26513,0],0] iof:hnp pushing fd 55 for process [[26513,1],3]
>> [c1725.crane.hcc.unl.edu <http://c1725.crane.hcc.unl.edu/>:218844] HCC 
>> debug: [[26513,0],0] iof:hnp pushing fd 45 for process [[26513,1],4]
>> [c1725.crane.hcc.unl.edu <http://c1725.crane.hcc.unl.edu/>:218844] HCC 
>> debug: [[26513,0],0] iof:hnp pushing fd 56 for process [[26513,1],4]
>> [c1725.crane.hcc.unl.edu <http://c1725.crane.hcc.unl.edu/>:218844] HCC 
>> debug: [[26513,0],0] iof:hnp pushing fd 58 for process [[26513,1],4]
>> [c1725.crane.hcc.unl.edu <http://c1725.crane.hcc.unl.edu/>:218844] HCC 
>> debug: [[26513,0],0] iof:hnp pushing fd 47 f

Re: [OMPI users] stdin issue with openmpi/2.0.0

[r...@open-mpi.org]

 pushing fd 70 for process [[26513,1],8]
> [c1725.crane.hcc.unl.edu <http://c1725.crane.hcc.unl.edu/>:218844] HCC debug: 
> [[26513,0],0] iof:hnp pushing fd 72 for process [[26513,1],8]
> [c1725.crane.hcc.unl.edu <http://c1725.crane.hcc.unl.edu/>:218844] HCC debug: 
> [[26513,0],0] iof:hnp pushing fd 67 for process [[26513,1],9]
> [c1725.crane.hcc.unl.edu <http://c1725.crane.hcc.unl.edu/>:218844] HCC debug: 
> [[26513,0],0] iof:hnp pushing fd 74 for process [[26513,1],9]
> [c1725.crane.hcc.unl.edu <http://c1725.crane.hcc.unl.edu/>:218844] HCC debug: 
> [[26513,0],0] iof:hnp pushing fd 76 for process [[26513,1],9]
> Rank 1 has cleared MPI_Init
> Rank 3 has cleared MPI_Init
> Rank 4 has cleared MPI_Init
> Rank 5 has cleared MPI_Init
> Rank 6 has cleared MPI_Init
> Rank 7 has cleared MPI_Init
> Rank 0 has cleared MPI_Init
> Rank 2 has cleared MPI_Init
> Rank 8 has cleared MPI_Init
> Rank 9 has cleared MPI_Init
> Rank 10 has cleared MPI_Init
> Rank 11 has cleared MPI_Init
> Rank 12 has cleared MPI_Init
> Rank 13 has cleared MPI_Init
> Rank 16 has cleared MPI_Init
> Rank 17 has cleared MPI_Init
> Rank 18 has cleared MPI_Init
> Rank 14 has cleared MPI_Init
> Rank 15 has cleared MPI_Init
> Rank 19 has cleared MPI_Init
> 
> 
> The part of code I changed in file ./orte/mca/iof/hnp/iof_hnp.c
> 
> opal_output(0,
>  "HCC debug: %s iof:hnp pushing fd %d for process %s",
>  ORTE_NAME_PRINT(ORTE_PROC_MY_NAME),
>  fd, ORTE_NAME_PRINT(dst_name));
> 
> /* don't do this if the dst vpid is invalid or the fd is negative! */
> if (ORTE_VPID_INVALID == dst_name->vpid || fd < 0) {
> return ORTE_SUCCESS;
> }
> 
> /*OPAL_OUTPUT_VERBOSE((1, orte_iof_base_framework.framework_output,
>  "%s iof:hnp pushing fd %d for process %s",
>  ORTE_NAME_PRINT(ORTE_PROC_MY_NAME),
>  fd, ORTE_NAME_PRINT(dst_name)));
> */
> 
> From: users <users-boun...@lists.open-mpi.org 
> <mailto:users-boun...@lists.open-mpi.org>> on behalf of r...@open-mpi.org 
> <mailto:r...@open-mpi.org> <r...@open-mpi.org <mailto:r...@open-mpi.org>>
> Sent: Monday, August 29, 2016 11:42:00 AM
> To: Open MPI Users
> Subject: Re: [OMPI users] stdin issue with openmpi/2.0.0
>  
> I’m sorry, but something is simply very wrong here. Are you sure you are 
> pointed at the correct LD_LIBRARY_PATH? Perhaps add a “BOO” or something at 
> the front of the output message to ensure we are using the correct plugin?
> 
> This looks to me like you must be picking up a stale library somewhere.
> 
>> On Aug 29, 2016, at 10:29 AM, Jingchao Zhang <zh...@unl.edu 
>> <mailto:zh...@unl.edu>> wrote:
>> 
>> Hi Ralph,
>> 
>> I used the tarball from Aug 26 and added the patch. Tested with 2 nodes, 10 
>> cores/node. Please see the results below:
>> 
>> $ mpirun ./a.out < test.in
>> [c1725.crane.hcc.unl.edu <http://c1725.crane.hcc.unl.edu/>:170750] 
>> [[43954,0],0] iof:hnp pushing fd 35 for process [[43954,1],0]
>> [c1725.crane.hcc.unl.edu <http://c1725.crane.hcc.unl.edu/>:170750] 
>> [[43954,0],0] iof:hnp pushing fd 41 for process [[43954,1],0]
>> [c1725.crane.hcc.unl.edu <http://c1725.crane.hcc.unl.edu/>:170750] 
>> [[43954,0],0] iof:hnp pushing fd 43 for process [[43954,1],0]
>> [c1725.crane.hcc.unl.edu <http://c1725.crane.hcc.unl.edu/>:170750] 
>> [[43954,0],0] iof:hnp pushing fd 37 for process [[43954,1],1]
>> [c1725.crane.hcc.unl.edu <http://c1725.crane.hcc.unl.edu/>:170750] 
>> [[43954,0],0] iof:hnp pushing fd 46 for process [[43954,1],1]
>> [c1725.crane.hcc.unl.edu <http://c1725.crane.hcc.unl.edu/>:170750] 
>> [[43954,0],0] iof:hnp pushing fd 49 for process [[43954,1],1]
>> [c1725.crane.hcc.unl.edu <http://c1725.crane.hcc.unl.edu/>:170750] 
>> [[43954,0],0] iof:hnp pushing fd 38 for process [[43954,1],2]
>> [c1725.crane.hcc.unl.edu <http://c1725.crane.hcc.unl.edu/>:170750] 
>> [[43954,0],0] iof:hnp pushing fd 50 for process [[43954,1],2]
>> [c1725.crane.hcc.unl.edu <http://c1725.crane.hcc.unl.edu/>:170750] 
>> [[43954,0],0] iof:hnp pushing fd 52 for process [[43954,1],2]
>> [c1725.crane.hcc.unl.edu <http://c1725.crane.hcc.unl.edu/>:170750] 
>> [[43954,0],0] iof:hnp pushing fd 42 for process [[43954,1],3]
>> [c1725.crane.hcc.unl.edu <http://c1725.crane.hcc.unl.edu/>:170750] 
>> [[43954,0],0] iof:hnp pushing fd 53 for process [[43954,1],3]
>> [c1725.crane.hcc.unl.edu <http://c1725.crane.hcc.un

Re: [OMPI users] stdin issue with openmpi/2.0.0

[r...@open-mpi.org]

nk 9 has cleared MPI_Init
> Rank 1 has cleared MPI_Init
> Rank 2 has cleared MPI_Init
> Rank 3 has cleared MPI_Init
> Rank 4 has cleared MPI_Init
> Rank 8 has cleared MPI_Init
> Rank 0 has cleared MPI_Init
> Rank 6 has cleared MPI_Init
> Rank 7 has cleared MPI_Init
> Rank 14 has cleared MPI_Init
> Rank 15 has cleared MPI_Init
> Rank 16 has cleared MPI_Init
> Rank 18 has cleared MPI_Init
> Rank 10 has cleared MPI_Init
> Rank 11 has cleared MPI_Init
> Rank 12 has cleared MPI_Init
> Rank 13 has cleared MPI_Init
> Rank 17 has cleared MPI_Init
> Rank 19 has cleared MPI_Init
> 
> Thanks,
> 
> Dr. Jingchao Zhang
> Holland Computing Center
> University of Nebraska-Lincoln
> 402-472-6400
> From: users <users-boun...@lists.open-mpi.org 
> <mailto:users-boun...@lists.open-mpi.org>> on behalf of r...@open-mpi.org 
> <mailto:r...@open-mpi.org> <r...@open-mpi.org <mailto:r...@open-mpi.org>>
> Sent: Saturday, August 27, 2016 12:31:53 PM
> To: Open MPI Users
> Subject: Re: [OMPI users] stdin issue with openmpi/2.0.0
>  
> I am finding this impossible to replicate, so something odd must be going on. 
> Can you please (a) pull down the latest v2.0.1 nightly tarball, and (b) add 
> this patch to it?
> 
> diff --git a/orte/mca/iof/hnp/iof_hnp.c b/orte/mca/iof/hnp/iof_hnp.c
> old mode 100644
> new mode 100755
> index 512fcdb..362ff46
> --- a/orte/mca/iof/hnp/iof_hnp.c
> +++ b/orte/mca/iof/hnp/iof_hnp.c
> @@ -143,16 +143,17 @@ static int hnp_push(const orte_process_name_t* 
> dst_name, orte_iof_tag_t src_tag,
>  int np, numdigs;
>  orte_ns_cmp_bitmask_t mask;
>  
> +opal_output(0,
> + "%s iof:hnp pushing fd %d for process %s",
> + ORTE_NAME_PRINT(ORTE_PROC_MY_NAME),
> + fd, ORTE_NAME_PRINT(dst_name));
> +
>  /* don't do this if the dst vpid is invalid or the fd is negative! */
>  if (ORTE_VPID_INVALID == dst_name->vpid || fd < 0) {
>  return ORTE_SUCCESS;
>  }
>  
> -OPAL_OUTPUT_VERBOSE((1, orte_iof_base_framework.framework_output,
> - "%s iof:hnp pushing fd %d for process %s",
> - ORTE_NAME_PRINT(ORTE_PROC_MY_NAME),
> - fd, ORTE_NAME_PRINT(dst_name)));
> -
>  if (!(src_tag & ORTE_IOF_STDIN)) {
>  /* set the file descriptor to non-blocking - do this before we setup
>   * and activate the read event in case it fires right away
> 
> 
> You can then run the test again without the "--mca iof_base_verbose 100” flag 
> to reduce the chatter - this print statement will tell me what I need to know.
> 
> Thanks!
> Ralph
> 
> 
>> On Aug 25, 2016, at 8:19 AM, Jeff Squyres (jsquyres) <jsquy...@cisco.com 
>> <mailto:jsquy...@cisco.com>> wrote:
>> 
>> The IOF fix PR for v2.0.1 was literally just merged a few minutes ago; it 
>> wasn't in last night's tarball.
>> 
>> 
>> 
>>> On Aug 25, 2016, at 10:59 AM, r...@open-mpi.org <mailto:r...@open-mpi.org> 
>>> wrote:
>>> 
>>> ??? Weird - can you send me an updated output of that last test we ran?
>>> 
>>>> On Aug 25, 2016, at 7:51 AM, Jingchao Zhang <zh...@unl.edu 
>>>> <mailto:zh...@unl.edu>> wrote:
>>>> 
>>>> Hi Ralph,
>>>> 
>>>> I saw the pull request and did a test with v2.0.1rc1, but the problem 
>>>> persists. Any ideas?
>>>> 
>>>> Thanks,
>>>> 
>>>> Dr. Jingchao Zhang
>>>> Holland Computing Center
>>>> University of Nebraska-Lincoln
>>>> 402-472-6400
>>>> From: users <users-boun...@lists.open-mpi.org 
>>>> <mailto:users-boun...@lists.open-mpi.org>> on behalf of r...@open-mpi.org 
>>>> <mailto:r...@open-mpi.org> <r...@open-mpi.org <mailto:r...@open-mpi.org>>
>>>> Sent: Wednesday, August 24, 2016 1:27:28 PM
>>>> To: Open MPI Users
>>>> Subject: Re: [OMPI users] stdin issue with openmpi/2.0.0
>>>> 
>>>> Bingo - found it, fix submitted and hope to get it into 2.0.1
>>>> 
>>>> Thanks for the assist!
>>>> Ralph
>>>> 
>>>> 
>>>>> On Aug 24, 2016, at 12:15 PM, Jingchao Zhang <zh...@unl.edu 
>>>>> <mailto:zh...@unl.edu>> wrote:
>>>>> 
>>>>> I configured v2.0.1rc1 with --enable-debug and ran the test with --mca 
>>>>> iof_base_verbose 100. I also added -display-dev

Re: [OMPI users] stdin issue with openmpi/2.0.0

[r...@open-mpi.org]

I am finding this impossible to replicate, so something odd must be going on. 
Can you please (a) pull down the latest v2.0.1 nightly tarball, and (b) add 
this patch to it?

diff --git a/orte/mca/iof/hnp/iof_hnp.c b/orte/mca/iof/hnp/iof_hnp.c
old mode 100644
new mode 100755
index 512fcdb..362ff46
--- a/orte/mca/iof/hnp/iof_hnp.c
+++ b/orte/mca/iof/hnp/iof_hnp.c
@@ -143,16 +143,17 @@ static int hnp_push(const orte_process_name_t* dst_name, 
orte_iof_tag_t src_tag,
 int np, numdigs;
 orte_ns_cmp_bitmask_t mask;
 
+opal_output(0,
+ "%s iof:hnp pushing fd %d for process %s",
+ ORTE_NAME_PRINT(ORTE_PROC_MY_NAME),
+ fd, ORTE_NAME_PRINT(dst_name));
+
 /* don't do this if the dst vpid is invalid or the fd is negative! */
 if (ORTE_VPID_INVALID == dst_name->vpid || fd < 0) {
 return ORTE_SUCCESS;
 }
 
-OPAL_OUTPUT_VERBOSE((1, orte_iof_base_framework.framework_output,
- "%s iof:hnp pushing fd %d for process %s",
- ORTE_NAME_PRINT(ORTE_PROC_MY_NAME),
- fd, ORTE_NAME_PRINT(dst_name)));
-
 if (!(src_tag & ORTE_IOF_STDIN)) {
 /* set the file descriptor to non-blocking - do this before we setup
  * and activate the read event in case it fires right away


You can then run the test again without the "--mca iof_base_verbose 100” flag 
to reduce the chatter - this print statement will tell me what I need to know.

Thanks!
Ralph


> On Aug 25, 2016, at 8:19 AM, Jeff Squyres (jsquyres) <jsquy...@cisco.com> 
> wrote:
> 
> The IOF fix PR for v2.0.1 was literally just merged a few minutes ago; it 
> wasn't in last night's tarball.
> 
> 
> 
>> On Aug 25, 2016, at 10:59 AM, r...@open-mpi.org wrote:
>> 
>> ??? Weird - can you send me an updated output of that last test we ran?
>> 
>>> On Aug 25, 2016, at 7:51 AM, Jingchao Zhang <zh...@unl.edu> wrote:
>>> 
>>> Hi Ralph,
>>> 
>>> I saw the pull request and did a test with v2.0.1rc1, but the problem 
>>> persists. Any ideas?
>>> 
>>> Thanks,
>>> 
>>> Dr. Jingchao Zhang
>>> Holland Computing Center
>>> University of Nebraska-Lincoln
>>> 402-472-6400
>>> From: users <users-boun...@lists.open-mpi.org> on behalf of 
>>> r...@open-mpi.org <r...@open-mpi.org>
>>> Sent: Wednesday, August 24, 2016 1:27:28 PM
>>> To: Open MPI Users
>>> Subject: Re: [OMPI users] stdin issue with openmpi/2.0.0
>>> 
>>> Bingo - found it, fix submitted and hope to get it into 2.0.1
>>> 
>>> Thanks for the assist!
>>> Ralph
>>> 
>>> 
>>>> On Aug 24, 2016, at 12:15 PM, Jingchao Zhang <zh...@unl.edu> wrote:
>>>> 
>>>> I configured v2.0.1rc1 with --enable-debug and ran the test with --mca 
>>>> iof_base_verbose 100. I also added -display-devel-map in case it provides 
>>>> some useful information.
>>>> 
>>>> Test job has 2 nodes, each node 10 cores. Rank 0 and mpirun command on the 
>>>> same node.
>>>> $ mpirun -display-devel-map --mca iof_base_verbose 100 ./a.out < test.in 
>>>> &> debug_info.txt
>>>> 
>>>> The debug_info.txt is attached. 
>>>> 
>>>> Dr. Jingchao Zhang
>>>> Holland Computing Center
>>>> University of Nebraska-Lincoln
>>>> 402-472-6400
>>>> From: users <users-boun...@lists.open-mpi.org> on behalf of 
>>>> r...@open-mpi.org <r...@open-mpi.org>
>>>> Sent: Wednesday, August 24, 2016 12:14:26 PM
>>>> To: Open MPI Users
>>>> Subject: Re: [OMPI users] stdin issue with openmpi/2.0.0
>>>> 
>>>> Afraid I can’t replicate a problem at all, whether rank=0 is local or not. 
>>>> I’m also using bash, but on CentOS-7, so I suspect the OS is the 
>>>> difference.
>>>> 
>>>> Can you configure OMPI with --enable-debug, and then run the test again 
>>>> with --mca iof_base_verbose 100? It will hopefully tell us something about 
>>>> why the IO subsystem is stuck.
>>>> 
>>>> 
>>>>> On Aug 24, 2016, at 8:46 AM, Jingchao Zhang <zh...@unl.edu> wrote:
>>>>> 
>>>>> Hi Ralph,
>>>>> 
>>>>> For our tests, rank 0 is always on the same node with mpirun. I just 
>>>>> tested mpirun with -nolocal and it still hangs. 
>>>>> 
>>>>> Information on shell and OS
>>>>>

Re: [OMPI users] stdin issue with openmpi/2.0.0

[r...@open-mpi.org]

??? Weird - can you send me an updated output of that last test we ran?

> On Aug 25, 2016, at 7:51 AM, Jingchao Zhang <zh...@unl.edu> wrote:
> 
> Hi Ralph,
> 
> I saw the pull request and did a test with v2.0.1rc1, but the problem 
> persists. Any ideas?
> 
> Thanks,
> 
> Dr. Jingchao Zhang
> Holland Computing Center
> University of Nebraska-Lincoln
> 402-472-6400
> From: users <users-boun...@lists.open-mpi.org 
> <mailto:users-boun...@lists.open-mpi.org>> on behalf of r...@open-mpi.org 
> <mailto:r...@open-mpi.org> <r...@open-mpi.org <mailto:r...@open-mpi.org>>
> Sent: Wednesday, August 24, 2016 1:27:28 PM
> To: Open MPI Users
> Subject: Re: [OMPI users] stdin issue with openmpi/2.0.0
>  
> Bingo - found it, fix submitted and hope to get it into 2.0.1
> 
> Thanks for the assist!
> Ralph
> 
> 
>> On Aug 24, 2016, at 12:15 PM, Jingchao Zhang <zh...@unl.edu 
>> <mailto:zh...@unl.edu>> wrote:
>> 
>> I configured v2.0.1rc1 with --enable-debug and ran the test with --mca 
>> iof_base_verbose 100. I also added -display-devel-map in case it provides 
>> some useful information.
>> 
>> Test job has 2 nodes, each node 10 cores. Rank 0 and mpirun command on the 
>> same node.
>> $ mpirun -display-devel-map --mca iof_base_verbose 100 ./a.out < test.in &> 
>> debug_info.txt
>> 
>> The debug_info.txt is attached. 
>> 
>> Dr. Jingchao Zhang
>> Holland Computing Center
>> University of Nebraska-Lincoln
>> 402-472-6400
>> From: users <users-boun...@lists.open-mpi.org 
>> <mailto:users-boun...@lists.open-mpi.org>> on behalf of r...@open-mpi.org 
>> <mailto:r...@open-mpi.org> <r...@open-mpi.org <mailto:r...@open-mpi.org>>
>> Sent: Wednesday, August 24, 2016 12:14:26 PM
>> To: Open MPI Users
>> Subject: Re: [OMPI users] stdin issue with openmpi/2.0.0
>>  
>> Afraid I can’t replicate a problem at all, whether rank=0 is local or not. 
>> I’m also using bash, but on CentOS-7, so I suspect the OS is the difference.
>> 
>> Can you configure OMPI with --enable-debug, and then run the test again with 
>> --mca iof_base_verbose 100? It will hopefully tell us something about why 
>> the IO subsystem is stuck.
>> 
>> 
>>> On Aug 24, 2016, at 8:46 AM, Jingchao Zhang <zh...@unl.edu 
>>> <mailto:zh...@unl.edu>> wrote:
>>> 
>>> Hi Ralph,
>>> 
>>> For our tests, rank 0 is always on the same node with mpirun. I just tested 
>>> mpirun with -nolocal and it still hangs. 
>>> 
>>> Information on shell and OS
>>> $ echo $0
>>> -bash
>>> 
>>> $ lsb_release -a
>>> LSB Version:
>>> :base-4.0-amd64:base-4.0-noarch:core-4.0-amd64:core-4.0-noarch:graphics-4.0-amd64:graphics-4.0-noarch:printing-4.0-amd64:printing-4.0-noarch
>>> Distributor ID: Scientific
>>> Description:Scientific Linux release 6.8 (Carbon)
>>> Release:6.8
>>> Codename:   Carbon
>>> 
>>> $ uname -a
>>> Linux login.crane.hcc.unl.edu <http://login.crane.hcc.unl.edu/> 
>>> 2.6.32-642.3.1.el6.x86_64 #1 SMP Tue Jul 12 11:25:51 CDT 2016 x86_64 x86_64 
>>> x86_64 GNU/Linux
>>> 
>>> 
>>> Dr. Jingchao Zhang
>>> Holland Computing Center
>>> University of Nebraska-Lincoln
>>> 402-472-6400
>>> From: users <users-boun...@lists.open-mpi.org 
>>> <mailto:users-boun...@lists.open-mpi.org>> on behalf of r...@open-mpi.org 
>>> <mailto:r...@open-mpi.org> <r...@open-mpi.org <mailto:r...@open-mpi.org>>
>>> Sent: Tuesday, August 23, 2016 8:14:48 PM
>>> To: Open MPI Users
>>> Subject: Re: [OMPI users] stdin issue with openmpi/2.0.0
>>>  
>>> Hmmm...that’s a good point. Rank 0 and mpirun are always on the same node 
>>> on my cluster. I’ll give it a try.
>>> 
>>> Jingchao: is rank 0 on the node with mpirun, or on a remote node?
>>> 
>>> 
>>>> On Aug 23, 2016, at 5:58 PM, Gilles Gouaillardet <gil...@rist.or.jp 
>>>> <mailto:gil...@rist.or.jp>> wrote:
>>>> 
>>>> Ralph,
>>>> 
>>>> did you run task 0 and mpirun on different nodes ?
>>>> 
>>>> i observed some random hangs, though i cannot blame openmpi 100% yet
>>>> 
>>>> Cheers,
>>>> 
>>>> Gilles
>>>> 
>>>> On 8/24/2016 9:41 AM, r...@open-mpi.org <mailto:r...@open-mpi.org> w

Re: [OMPI users] stdin issue with openmpi/2.0.0

[r...@open-mpi.org]

Bingo - found it, fix submitted and hope to get it into 2.0.1

Thanks for the assist!
Ralph


> On Aug 24, 2016, at 12:15 PM, Jingchao Zhang <zh...@unl.edu> wrote:
> 
> I configured v2.0.1rc1 with --enable-debug and ran the test with --mca 
> iof_base_verbose 100. I also added -display-devel-map in case it provides 
> some useful information.
> 
> Test job has 2 nodes, each node 10 cores. Rank 0 and mpirun command on the 
> same node.
> $ mpirun -display-devel-map --mca iof_base_verbose 100 ./a.out < test.in &> 
> debug_info.txt
> 
> The debug_info.txt is attached. 
> 
> Dr. Jingchao Zhang
> Holland Computing Center
> University of Nebraska-Lincoln
> 402-472-6400
> From: users <users-boun...@lists.open-mpi.org 
> <mailto:users-boun...@lists.open-mpi.org>> on behalf of r...@open-mpi.org 
> <mailto:r...@open-mpi.org> <r...@open-mpi.org <mailto:r...@open-mpi.org>>
> Sent: Wednesday, August 24, 2016 12:14:26 PM
> To: Open MPI Users
> Subject: Re: [OMPI users] stdin issue with openmpi/2.0.0
>  
> Afraid I can’t replicate a problem at all, whether rank=0 is local or not. 
> I’m also using bash, but on CentOS-7, so I suspect the OS is the difference.
> 
> Can you configure OMPI with --enable-debug, and then run the test again with 
> --mca iof_base_verbose 100? It will hopefully tell us something about why the 
> IO subsystem is stuck.
> 
> 
>> On Aug 24, 2016, at 8:46 AM, Jingchao Zhang <zh...@unl.edu 
>> <mailto:zh...@unl.edu>> wrote:
>> 
>> Hi Ralph,
>> 
>> For our tests, rank 0 is always on the same node with mpirun. I just tested 
>> mpirun with -nolocal and it still hangs. 
>> 
>> Information on shell and OS
>> $ echo $0
>> -bash
>> 
>> $ lsb_release -a
>> LSB Version:
>> :base-4.0-amd64:base-4.0-noarch:core-4.0-amd64:core-4.0-noarch:graphics-4.0-amd64:graphics-4.0-noarch:printing-4.0-amd64:printing-4.0-noarch
>> Distributor ID: Scientific
>> Description:Scientific Linux release 6.8 (Carbon)
>> Release:6.8
>> Codename:   Carbon
>> 
>> $ uname -a
>> Linux login.crane.hcc.unl.edu <http://login.crane.hcc.unl.edu/> 
>> 2.6.32-642.3.1.el6.x86_64 #1 SMP Tue Jul 12 11:25:51 CDT 2016 x86_64 x86_64 
>> x86_64 GNU/Linux
>> 
>> 
>> Dr. Jingchao Zhang
>> Holland Computing Center
>> University of Nebraska-Lincoln
>> 402-472-6400
>> From: users <users-boun...@lists.open-mpi.org 
>> <mailto:users-boun...@lists.open-mpi.org>> on behalf of r...@open-mpi.org 
>> <mailto:r...@open-mpi.org> <r...@open-mpi.org <mailto:r...@open-mpi.org>>
>> Sent: Tuesday, August 23, 2016 8:14:48 PM
>> To: Open MPI Users
>> Subject: Re: [OMPI users] stdin issue with openmpi/2.0.0
>>  
>> Hmmm...that’s a good point. Rank 0 and mpirun are always on the same node on 
>> my cluster. I’ll give it a try.
>> 
>> Jingchao: is rank 0 on the node with mpirun, or on a remote node?
>> 
>> 
>>> On Aug 23, 2016, at 5:58 PM, Gilles Gouaillardet <gil...@rist.or.jp 
>>> <mailto:gil...@rist.or.jp>> wrote:
>>> 
>>> Ralph,
>>> 
>>> did you run task 0 and mpirun on different nodes ?
>>> 
>>> i observed some random hangs, though i cannot blame openmpi 100% yet
>>> 
>>> Cheers,
>>> 
>>> Gilles
>>> 
>>> On 8/24/2016 9:41 AM, r...@open-mpi.org <mailto:r...@open-mpi.org> wrote:
>>>> Very strange. I cannot reproduce it as I’m able to run any number of nodes 
>>>> and procs, pushing over 100Mbytes thru without any problem.
>>>> 
>>>> Which leads me to suspect that the issue here is with the tty interface. 
>>>> Can you tell me what shell and OS you are running?
>>>> 
>>>> 
>>>>> On Aug 23, 2016, at 3:25 PM, Jingchao Zhang <zh...@unl.edu 
>>>>> <mailto:zh...@unl.edu>> wrote:
>>>>> 
>>>>> Everything stuck at MPI_Init. For a test job with 2 nodes and 10 cores 
>>>>> each node, I got the following
>>>>> 
>>>>> $ mpirun ./a.out < test.in
>>>>> Rank 2 has cleared MPI_Init
>>>>> Rank 4 has cleared MPI_Init
>>>>> Rank 7 has cleared MPI_Init
>>>>> Rank 8 has cleared MPI_Init
>>>>> Rank 0 has cleared MPI_Init
>>>>> Rank 5 has cleared MPI_Init
>>>>> Rank 6 has cleared MPI_Init
>>>>> Rank 9 has cleared MPI_Init
>>>>> Rank 1 has cleared MPI_Init

Re: [OMPI users] stdin issue with openmpi/2.0.0

[r...@open-mpi.org]

Afraid I can’t replicate a problem at all, whether rank=0 is local or not. I’m 
also using bash, but on CentOS-7, so I suspect the OS is the difference.

Can you configure OMPI with --enable-debug, and then run the test again with 
--mca iof_base_verbose 100? It will hopefully tell us something about why the 
IO subsystem is stuck.


> On Aug 24, 2016, at 8:46 AM, Jingchao Zhang <zh...@unl.edu> wrote:
> 
> Hi Ralph,
> 
> For our tests, rank 0 is always on the same node with mpirun. I just tested 
> mpirun with -nolocal and it still hangs. 
> 
> Information on shell and OS
> $ echo $0
> -bash
> 
> $ lsb_release -a
> LSB Version:
> :base-4.0-amd64:base-4.0-noarch:core-4.0-amd64:core-4.0-noarch:graphics-4.0-amd64:graphics-4.0-noarch:printing-4.0-amd64:printing-4.0-noarch
> Distributor ID: Scientific
> Description:Scientific Linux release 6.8 (Carbon)
> Release:6.8
> Codename:   Carbon
> 
> $ uname -a
> Linux login.crane.hcc.unl.edu <http://login.crane.hcc.unl.edu/> 
> 2.6.32-642.3.1.el6.x86_64 #1 SMP Tue Jul 12 11:25:51 CDT 2016 x86_64 x86_64 
> x86_64 GNU/Linux
> 
> 
> Dr. Jingchao Zhang
> Holland Computing Center
> University of Nebraska-Lincoln
> 402-472-6400
> From: users <users-boun...@lists.open-mpi.org 
> <mailto:users-boun...@lists.open-mpi.org>> on behalf of r...@open-mpi.org 
> <mailto:r...@open-mpi.org> <r...@open-mpi.org <mailto:r...@open-mpi.org>>
> Sent: Tuesday, August 23, 2016 8:14:48 PM
> To: Open MPI Users
> Subject: Re: [OMPI users] stdin issue with openmpi/2.0.0
>  
> Hmmm...that’s a good point. Rank 0 and mpirun are always on the same node on 
> my cluster. I’ll give it a try.
> 
> Jingchao: is rank 0 on the node with mpirun, or on a remote node?
> 
> 
>> On Aug 23, 2016, at 5:58 PM, Gilles Gouaillardet <gil...@rist.or.jp 
>> <mailto:gil...@rist.or.jp>> wrote:
>> 
>> Ralph,
>> 
>> did you run task 0 and mpirun on different nodes ?
>> 
>> i observed some random hangs, though i cannot blame openmpi 100% yet
>> 
>> Cheers,
>> 
>> Gilles
>> 
>> On 8/24/2016 9:41 AM, r...@open-mpi.org <mailto:r...@open-mpi.org> wrote:
>>> Very strange. I cannot reproduce it as I’m able to run any number of nodes 
>>> and procs, pushing over 100Mbytes thru without any problem.
>>> 
>>> Which leads me to suspect that the issue here is with the tty interface. 
>>> Can you tell me what shell and OS you are running?
>>> 
>>> 
>>>> On Aug 23, 2016, at 3:25 PM, Jingchao Zhang <zh...@unl.edu 
>>>> <mailto:zh...@unl.edu>> wrote:
>>>> 
>>>> Everything stuck at MPI_Init. For a test job with 2 nodes and 10 cores 
>>>> each node, I got the following
>>>> 
>>>> $ mpirun ./a.out < test.in
>>>> Rank 2 has cleared MPI_Init
>>>> Rank 4 has cleared MPI_Init
>>>> Rank 7 has cleared MPI_Init
>>>> Rank 8 has cleared MPI_Init
>>>> Rank 0 has cleared MPI_Init
>>>> Rank 5 has cleared MPI_Init
>>>> Rank 6 has cleared MPI_Init
>>>> Rank 9 has cleared MPI_Init
>>>> Rank 1 has cleared MPI_Init
>>>> Rank 16 has cleared MPI_Init
>>>> Rank 19 has cleared MPI_Init
>>>> Rank 10 has cleared MPI_Init
>>>> Rank 11 has cleared MPI_Init
>>>> Rank 12 has cleared MPI_Init
>>>> Rank 13 has cleared MPI_Init
>>>> Rank 14 has cleared MPI_Init
>>>> Rank 15 has cleared MPI_Init
>>>> Rank 17 has cleared MPI_Init
>>>> Rank 18 has cleared MPI_Init
>>>> Rank 3 has cleared MPI_Init
>>>> 
>>>> then it just hanged.
>>>> 
>>>> --Jingchao
>>>> 
>>>> Dr. Jingchao Zhang
>>>> Holland Computing Center
>>>> University of Nebraska-Lincoln
>>>> 402-472-6400
>>>> From: users <users-boun...@lists.open-mpi.org 
>>>> <mailto:users-boun...@lists.open-mpi.org>> on behalf of r...@open-mpi.org 
>>>> <mailto:r...@open-mpi.org> <r...@open-mpi.org <mailto:r...@open-mpi.org>>
>>>> Sent: Tuesday, August 23, 2016 4:03:07 PM
>>>> To: Open MPI Users
>>>> Subject: Re: [OMPI users] stdin issue with openmpi/2.0.0
>>>>  
>>>> The IO forwarding messages all flow over the Ethernet, so the type of 
>>>> fabric is irrelevant. The number of procs involved would definitely have 
>>>> an impact, but that might not be due to the IO forwarding subsystem. We 
>

Re: [OMPI users] Using Open MPI with PBS Pro

[r...@open-mpi.org]

Ah yes - that was a case where someone had added an incorrect param to the 
default file, and we suggested they remove it


> On Aug 24, 2016, at 5:26 AM, Andy Riebs <andy.ri...@hpe.com> wrote:
> 
> Hi Ralph,
> 
> I think I found that information at 
> <https://github.com/open-mpi/ompi/issues/341> :-)
> 
> In any case, thanks for the information about the default params file -- I 
> won't worry too much about modifying it then.
> 
> Andy
> 
> I
> On 08/23/2016 08:08 PM, r...@open-mpi.org wrote:
>> I’ve never heard of that, and cannot imagine what it has to do with the 
>> resource manager. Can you point to where you heard that one?
>> 
>> FWIW: we don’t ship OMPI with anything in the default mca params file, so 
>> somebody must have put it in there for you.
>> 
>> 
>>> On Aug 23, 2016, at 4:48 PM, Andy Riebs <andy.ri...@hpe.com> wrote:
>>> 
>>> I gleaned from the web that I need to comment out 
>>> "opal_event_include=epoll" in /etc/openmpi-mca-params.conf in 
>>> order to use Open MPI with PBS Pro.
>>> 
>>> Can we also disable that in other cases, like Slurm, or is this something 
>>> specific to PBS Pro?
>>> 
>>> Andy
>>> 
>>> -- 
>>> Andy Riebs
>>> andy.ri...@hpe.com
>>> Hewlett-Packard Enterprise
>>> High Performance Computing Software Engineering
>>> +1 404 648 9024
>>> My opinions are not necessarily those of HPE
>>>May the source be with you!
>>> 
>>> ___
>>> users mailing list
>>> users@lists.open-mpi.org
>>> https://rfd.newmexicoconsortium.org/mailman/listinfo/users
>> ___
>> users mailing list
>> users@lists.open-mpi.org
>> https://rfd.newmexicoconsortium.org/mailman/listinfo/users
> 
> ___
> users mailing list
> users@lists.open-mpi.org
> https://rfd.newmexicoconsortium.org/mailman/listinfo/users

___
users mailing list
users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

Re: [OMPI users] stdin issue with openmpi/2.0.0

[r...@open-mpi.org]

Very strange. I cannot reproduce it as I’m able to run any number of nodes and 
procs, pushing over 100Mbytes thru without any problem.

Which leads me to suspect that the issue here is with the tty interface. Can 
you tell me what shell and OS you are running?


> On Aug 23, 2016, at 3:25 PM, Jingchao Zhang <zh...@unl.edu> wrote:
> 
> Everything stuck at MPI_Init. For a test job with 2 nodes and 10 cores each 
> node, I got the following
> 
> $ mpirun ./a.out < test.in
> Rank 2 has cleared MPI_Init
> Rank 4 has cleared MPI_Init
> Rank 7 has cleared MPI_Init
> Rank 8 has cleared MPI_Init
> Rank 0 has cleared MPI_Init
> Rank 5 has cleared MPI_Init
> Rank 6 has cleared MPI_Init
> Rank 9 has cleared MPI_Init
> Rank 1 has cleared MPI_Init
> Rank 16 has cleared MPI_Init
> Rank 19 has cleared MPI_Init
> Rank 10 has cleared MPI_Init
> Rank 11 has cleared MPI_Init
> Rank 12 has cleared MPI_Init
> Rank 13 has cleared MPI_Init
> Rank 14 has cleared MPI_Init
> Rank 15 has cleared MPI_Init
> Rank 17 has cleared MPI_Init
> Rank 18 has cleared MPI_Init
> Rank 3 has cleared MPI_Init
> 
> then it just hanged.
> 
> --Jingchao
> 
> Dr. Jingchao Zhang
> Holland Computing Center
> University of Nebraska-Lincoln
> 402-472-6400
> From: users <users-boun...@lists.open-mpi.org 
> <mailto:users-boun...@lists.open-mpi.org>> on behalf of r...@open-mpi.org 
> <mailto:r...@open-mpi.org> <r...@open-mpi.org <mailto:r...@open-mpi.org>>
> Sent: Tuesday, August 23, 2016 4:03:07 PM
> To: Open MPI Users
> Subject: Re: [OMPI users] stdin issue with openmpi/2.0.0
>  
> The IO forwarding messages all flow over the Ethernet, so the type of fabric 
> is irrelevant. The number of procs involved would definitely have an impact, 
> but that might not be due to the IO forwarding subsystem. We know we have 
> flow control issues with collectives like Bcast that don’t have built-in 
> synchronization points. How many reads were you able to do before it hung?
> 
> I was running it on my little test setup (2 nodes, using only a few procs), 
> but I’ll try scaling up and see what happens. I’ll also try introducing some 
> forced “syncs” on the Bcast and see if that solves the issue.
> 
> Ralph
> 
>> On Aug 23, 2016, at 2:30 PM, Jingchao Zhang <zh...@unl.edu 
>> <mailto:zh...@unl.edu>> wrote:
>> 
>> Hi Ralph,
>> 
>> I tested v2.0.1rc1 with your code but has the same issue. I also installed 
>> v2.0.1rc1 on a different cluster which has Mellanox QDR Infiniband and get 
>> the same result. For the tests you have done, how many cores and nodes did 
>> you use? I can trigger the problem by using multiple nodes and each node 
>> with more than 10 cores. 
>> 
>> Thank you for looking into this.
>> 
>> Dr. Jingchao Zhang
>> Holland Computing Center
>> University of Nebraska-Lincoln
>> 402-472-6400
>> From: users <users-boun...@lists.open-mpi.org 
>> <mailto:users-boun...@lists.open-mpi.org>> on behalf of r...@open-mpi.org 
>> <mailto:r...@open-mpi.org> <r...@open-mpi.org <mailto:r...@open-mpi.org>>
>> Sent: Monday, August 22, 2016 10:23:42 PM
>> To: Open MPI Users
>> Subject: Re: [OMPI users] stdin issue with openmpi/2.0.0
>>  
>> FWIW: I just tested forwarding up to 100MBytes via stdin using the simple 
>> test shown below with OMPI v2.0.1rc1, and it worked fine. So I’d suggest 
>> upgrading when the official release comes out, or going ahead and at least 
>> testing 2.0.1rc1 on your machine. Or you can test this program with some 
>> input file and let me know if it works for you.
>> 
>> Ralph
>> 
>> #include 
>> #include 
>> #include 
>> #include 
>> #include 
>> #include 
>> 
>> #define ORTE_IOF_BASE_MSG_MAX   2048
>> 
>> int main(int argc, char *argv[])
>> {
>> int i, rank, size, next, prev, tag = 201;
>> int pos, msgsize, nbytes;
>> bool done;
>> char *msg;
>> 
>> MPI_Init(, );
>> MPI_Comm_rank(MPI_COMM_WORLD, );
>> MPI_Comm_size(MPI_COMM_WORLD, );
>> 
>> fprintf(stderr, "Rank %d has cleared MPI_Init\n", rank);
>> 
>> next = (rank + 1) % size;
>> prev = (rank + size - 1) % size;
>> msg = malloc(ORTE_IOF_BASE_MSG_MAX);
>> pos = 0;
>> nbytes = 0;
>> 
>> if (0 == rank) {
>> while (0 != (msgsize = read(0, msg, ORTE_IOF_BASE_MSG_MAX))) {
>> fprintf(stderr, "Rank %d: sending blob %d\n", rank, pos);
>> if (msgsize > 0) {
>>   

Re: [OMPI users] Using Open MPI with PBS Pro

[r...@open-mpi.org]

I’ve never heard of that, and cannot imagine what it has to do with the 
resource manager. Can you point to where you heard that one?

FWIW: we don’t ship OMPI with anything in the default mca params file, so 
somebody must have put it in there for you.


> On Aug 23, 2016, at 4:48 PM, Andy Riebs  wrote:
> 
> I gleaned from the web that I need to comment out "opal_event_include=epoll" 
> in /etc/openmpi-mca-params.conf in order to use Open MPI with 
> PBS Pro.
> 
> Can we also disable that in other cases, like Slurm, or is this something 
> specific to PBS Pro?
> 
> Andy
> 
> -- 
> Andy Riebs
> andy.ri...@hpe.com
> Hewlett-Packard Enterprise
> High Performance Computing Software Engineering
> +1 404 648 9024
> My opinions are not necessarily those of HPE
>May the source be with you!
> 
> ___
> users mailing list
> users@lists.open-mpi.org
> https://rfd.newmexicoconsortium.org/mailman/listinfo/users

___
users mailing list
users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

Re: [OMPI users] stdin issue with openmpi/2.0.0

[r...@open-mpi.org]

The IO forwarding messages all flow over the Ethernet, so the type of fabric is 
irrelevant. The number of procs involved would definitely have an impact, but 
that might not be due to the IO forwarding subsystem. We know we have flow 
control issues with collectives like Bcast that don’t have built-in 
synchronization points. How many reads were you able to do before it hung?

I was running it on my little test setup (2 nodes, using only a few procs), but 
I’ll try scaling up and see what happens. I’ll also try introducing some forced 
“syncs” on the Bcast and see if that solves the issue.

Ralph

> On Aug 23, 2016, at 2:30 PM, Jingchao Zhang <zh...@unl.edu> wrote:
> 
> Hi Ralph,
> 
> I tested v2.0.1rc1 with your code but has the same issue. I also installed 
> v2.0.1rc1 on a different cluster which has Mellanox QDR Infiniband and get 
> the same result. For the tests you have done, how many cores and nodes did 
> you use? I can trigger the problem by using multiple nodes and each node with 
> more than 10 cores. 
> 
> Thank you for looking into this.
> 
> Dr. Jingchao Zhang
> Holland Computing Center
> University of Nebraska-Lincoln
> 402-472-6400
> From: users <users-boun...@lists.open-mpi.org 
> <mailto:users-boun...@lists.open-mpi.org>> on behalf of r...@open-mpi.org 
> <mailto:r...@open-mpi.org> <r...@open-mpi.org <mailto:r...@open-mpi.org>>
> Sent: Monday, August 22, 2016 10:23:42 PM
> To: Open MPI Users
> Subject: Re: [OMPI users] stdin issue with openmpi/2.0.0
>  
> FWIW: I just tested forwarding up to 100MBytes via stdin using the simple 
> test shown below with OMPI v2.0.1rc1, and it worked fine. So I’d suggest 
> upgrading when the official release comes out, or going ahead and at least 
> testing 2.0.1rc1 on your machine. Or you can test this program with some 
> input file and let me know if it works for you.
> 
> Ralph
> 
> #include 
> #include 
> #include 
> #include 
> #include 
> #include 
> 
> #define ORTE_IOF_BASE_MSG_MAX   2048
> 
> int main(int argc, char *argv[])
> {
> int i, rank, size, next, prev, tag = 201;
> int pos, msgsize, nbytes;
> bool done;
> char *msg;
> 
> MPI_Init(, );
> MPI_Comm_rank(MPI_COMM_WORLD, );
> MPI_Comm_size(MPI_COMM_WORLD, );
> 
> fprintf(stderr, "Rank %d has cleared MPI_Init\n", rank);
> 
> next = (rank + 1) % size;
> prev = (rank + size - 1) % size;
> msg = malloc(ORTE_IOF_BASE_MSG_MAX);
> pos = 0;
> nbytes = 0;
> 
> if (0 == rank) {
> while (0 != (msgsize = read(0, msg, ORTE_IOF_BASE_MSG_MAX))) {
> fprintf(stderr, "Rank %d: sending blob %d\n", rank, pos);
> if (msgsize > 0) {
> MPI_Bcast(msg, ORTE_IOF_BASE_MSG_MAX, MPI_BYTE, 0, 
> MPI_COMM_WORLD);
> }
> ++pos;
> nbytes += msgsize;
> }
> fprintf(stderr, "Rank %d: sending termination blob %d\n", rank, pos);
> memset(msg, 0, ORTE_IOF_BASE_MSG_MAX);
> MPI_Bcast(msg, ORTE_IOF_BASE_MSG_MAX, MPI_BYTE, 0, MPI_COMM_WORLD);
> MPI_Barrier(MPI_COMM_WORLD);
> } else {
> while (1) {
> MPI_Bcast(msg, ORTE_IOF_BASE_MSG_MAX, MPI_BYTE, 0, 
> MPI_COMM_WORLD);
> fprintf(stderr, "Rank %d: recvd blob %d\n", rank, pos);
> ++pos;
> done = true;
> for (i=0; i < ORTE_IOF_BASE_MSG_MAX; i++) {
> if (0 != msg[i]) {
> done = false;
> break;
> }
> }
> if (done) {
> break;
> }
> }
> fprintf(stderr, "Rank %d: recv done\n", rank);
> MPI_Barrier(MPI_COMM_WORLD);
> }
> 
> fprintf(stderr, "Rank %d has completed bcast\n", rank);
> MPI_Finalize();
> return 0;
> }
> 
> 
> 
>> On Aug 22, 2016, at 3:40 PM, Jingchao Zhang <zh...@unl.edu 
>> <mailto:zh...@unl.edu>> wrote:
>> 
>> This might be a thin argument but we have many users running mpirun in this 
>> way for years with no problem until this recent upgrade. And some 
>> home-brewed mpi codes do not even have a standard way to read the input 
>> files. Last time I checked, the openmpi manual still claims it supports 
>> stdin (https://www.open-mpi.org/doc/v2.0/man1/mpirun.1.php#sect14 
>> <https://www.open-mpi.org/doc/v2.0/man1/mpirun.1.php#sect14>). Maybe I 
>> missed it but the v2.0 release notes did not mention any changes to the 
>> behaviors of stdin as well.
>> 
>> We can tell our users

Re: [OMPI users] OS X El Capitan 10.11.6 ld: symbol(s) not found for architecture x86_64

[r...@open-mpi.org]

I’m confused - you keep talking about MPICH, but the symbol you are looking for 
is from OMPI. You cannot mix the two MPI libraries - is that what you are 
trying to do?

> On Aug 23, 2016, at 1:30 PM, Richard G French  wrote:
> 
> Thanks for the suggestion, Doug - but I can't seem to find the missing 
> function ompi_mpi_byte in any of those other libraries. I'll keep looking! I 
> wonder if I failed to configure mpich properly when I built it.
> Dick
> 
> 
> On Tue, Aug 23, 2016 at 4:01 PM, Douglas L Reeder  > wrote:
> Richard,
> 
> It looks like you need to add some -l arguments to the the specific 
> openmpi libraries hat you need (e.g., -lmpi -lmpi_cxx)
> 
> Doug
>> On Aug 23, 2016, at 1:43 PM, Richard G French > > wrote:
>> 
>> Hi, all -
>> I'm trying to build the SPH code Gadget2 
>> (http://wwwmpa.mpa-garching.mpg.de/gadget/ 
>> ) under OS X 10.11.6 and I am 
>> getting the following type of error:
>> 
>> 222 rfrench@cosmos> make
>> 
>> mpicc main.o  run.o  predict.o begrun.o endrun.o global.o timestep.o  init.o 
>> restart.o  io.o accel.o   read_ic.o  ngb.o system.o  allocate.o  density.o 
>> gravtree.o hydra.o  driftfac.o domain.o  allvars.o potential.o forcetree.o   
>> peano.o gravtree_forcetest.o pm_periodic.o pm_nonperiodic.o longrange.o   -g 
>>  -L/opt/local/lib/mpich-mp  -L/usr/local/lib -lgsl -lgslcblas -lm 
>> -L/usr/local/lib -lrfftw_mpi -lfftw_mpi -lrfftw -lfftw-o  Gadget2  
>> 
>> Undefined symbols for architecture x86_64:
>> 
>>   "_ompi_mpi_byte", referenced from:
>> 
>>   _read_parameter_file in begrun.o
>> 
>>   _compute_global_quantities_of_system in global.o
>> 
>>   _restart in restart.o
>> 
>>   _write_file in io.o
>> 
>>   _read_file in read_ic.o
>> 
>>   _find_files in read_ic.o
>> 
>>   _density in density.o
>> 
>> ..
>> 
>> I built the mpich library using 
>> 
>> cd openmpi-2.0.0/
>> 
>> 
>> ./configure
>> 
>> 
>> sudo make all install
>> 
>> which installed the libraries in
>> 
>> 
>> /opt/local/lib/mpich-mp
>> 
>> 
>> 
>> I can't seem to track down the library that contains ompi_mpi_byte.
>> 
>> 
>> 
>> Any suggestions would be welcome. Thanks!
>> 
>> Dick French
>> 
>> 
>> 
>> 
>> -- 
>> Richard G. French
>> McDowell and Whiting Professor of Astrophysics
>> Chair of the Astronomy Department, Wellesley College
>> Director of the Whitin Observatory
>> Cassini Mission to Saturn Radio Science Team Leader
>> Wellesley, MA 02481-8203
>> (781) 283-3747 
>> ___
>> users mailing list
>> users@lists.open-mpi.org 
>> https://rfd.newmexicoconsortium.org/mailman/listinfo/users 
>> 
> 
> ___
> users mailing list
> users@lists.open-mpi.org 
> https://rfd.newmexicoconsortium.org/mailman/listinfo/users 
> 
> 
> 
> 
> -- 
> Richard G. French
> McDowell and Whiting Professor of Astrophysics
> Chair of the Astronomy Department, Wellesley College
> Director of the Whitin Observatory
> Cassini Mission to Saturn Radio Science Team Leader
> Wellesley, MA 02481-8203
> (781) 283-3747
> ___
> users mailing list
> users@lists.open-mpi.org
> https://rfd.newmexicoconsortium.org/mailman/listinfo/users

___
users mailing list
users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

Re: [OMPI users] stdin issue with openmpi/2.0.0

[r...@open-mpi.org]

FWIW: I just tested forwarding up to 100MBytes via stdin using the simple test 
shown below with OMPI v2.0.1rc1, and it worked fine. So I’d suggest upgrading 
when the official release comes out, or going ahead and at least testing 
2.0.1rc1 on your machine. Or you can test this program with some input file and 
let me know if it works for you.

Ralph

#include 
#include 
#include 
#include 
#include 
#include 

#define ORTE_IOF_BASE_MSG_MAX   2048

int main(int argc, char *argv[])
{
int i, rank, size, next, prev, tag = 201;
int pos, msgsize, nbytes;
bool done;
char *msg;

MPI_Init(, );
MPI_Comm_rank(MPI_COMM_WORLD, );
MPI_Comm_size(MPI_COMM_WORLD, );

fprintf(stderr, "Rank %d has cleared MPI_Init\n", rank);

next = (rank + 1) % size;
prev = (rank + size - 1) % size;
msg = malloc(ORTE_IOF_BASE_MSG_MAX);
pos = 0;
nbytes = 0;

if (0 == rank) {
while (0 != (msgsize = read(0, msg, ORTE_IOF_BASE_MSG_MAX))) {
fprintf(stderr, "Rank %d: sending blob %d\n", rank, pos);
if (msgsize > 0) {
MPI_Bcast(msg, ORTE_IOF_BASE_MSG_MAX, MPI_BYTE, 0, 
MPI_COMM_WORLD);
}
++pos;
nbytes += msgsize;
}
fprintf(stderr, "Rank %d: sending termination blob %d\n", rank, pos);
memset(msg, 0, ORTE_IOF_BASE_MSG_MAX);
MPI_Bcast(msg, ORTE_IOF_BASE_MSG_MAX, MPI_BYTE, 0, MPI_COMM_WORLD);
MPI_Barrier(MPI_COMM_WORLD);
} else {
while (1) {
MPI_Bcast(msg, ORTE_IOF_BASE_MSG_MAX, MPI_BYTE, 0, MPI_COMM_WORLD);
fprintf(stderr, "Rank %d: recvd blob %d\n", rank, pos);
++pos;
done = true;
for (i=0; i < ORTE_IOF_BASE_MSG_MAX; i++) {
if (0 != msg[i]) {
done = false;
break;
}
}
if (done) {
break;
}
}
fprintf(stderr, "Rank %d: recv done\n", rank);
MPI_Barrier(MPI_COMM_WORLD);
}

fprintf(stderr, "Rank %d has completed bcast\n", rank);
MPI_Finalize();
return 0;
}



> On Aug 22, 2016, at 3:40 PM, Jingchao Zhang <zh...@unl.edu> wrote:
> 
> This might be a thin argument but we have many users running mpirun in this 
> way for years with no problem until this recent upgrade. And some home-brewed 
> mpi codes do not even have a standard way to read the input files. Last time 
> I checked, the openmpi manual still claims it supports stdin 
> (https://www.open-mpi.org/doc/v2.0/man1/mpirun.1.php#sect14 
> <https://www.open-mpi.org/doc/v2.0/man1/mpirun.1.php#sect14>). Maybe I missed 
> it but the v2.0 release notes did not mention any changes to the behaviors of 
> stdin as well.
> 
> We can tell our users to run mpirun in the suggested way, but I do hope 
> someone can look into the issue and fix it.
> 
> Dr. Jingchao Zhang
> Holland Computing Center
> University of Nebraska-Lincoln
> 402-472-6400
> From: users <users-boun...@lists.open-mpi.org 
> <mailto:users-boun...@lists.open-mpi.org>> on behalf of r...@open-mpi.org 
> <mailto:r...@open-mpi.org> <r...@open-mpi.org <mailto:r...@open-mpi.org>>
> Sent: Monday, August 22, 2016 3:04:50 PM
> To: Open MPI Users
> Subject: Re: [OMPI users] stdin issue with openmpi/2.0.0
>  
> Well, I can try to find time to take a look. However, I will reiterate what 
> Jeff H said - it is very unwise to rely on IO forwarding. Much better to just 
> directly read the file unless that file is simply unavailable on the node 
> where rank=0 is running.
> 
>> On Aug 22, 2016, at 1:55 PM, Jingchao Zhang <zh...@unl.edu 
>> <mailto:zh...@unl.edu>> wrote:
>> 
>> Here you can find the source code for lammps input 
>> https://github.com/lammps/lammps/blob/r13864/src/input.cpp 
>> <https://github.com/lammps/lammps/blob/r13864/src/input.cpp>
>> Based on the gdb output, rank 0 stuck at line 167
>> if
>>  (fgets([m],maxline-m,infile)
>>  == NULL)
>> and the rest threads stuck at line 203
>> MPI_Bcast(,1,MPI_INT,0,world);
>> 
>> So rank 0 possibly hangs on the fgets() function.
>> 
>> Here are the whole backtrace information:
>> $ cat master.backtrace worker.backtrace
>> #0  0x003c37cdb68d in read () from /lib64/libc.so.6
>> #1  0x003c37c71ca8 in _IO_new_file_underflow () from /lib64/libc.so.6
>> #2  0x003c37c737ae in _IO_default_uflow_internal () from /lib64/libc.so.6
>> #3  0x003c37c67e8a in _IO_getline_info_internal () from /lib64/libc.so.6
>> #4  0x003c37c66ce9 in fgets () from /lib64/libc.so.6
>> #5  0x005c5a43 in LAMMPS_NS::Input::file() () at ../input.cpp

Re: [OMPI users] stdin issue with openmpi/2.0.0

[r...@open-mpi.org]

Well, I can try to find time to take a look. However, I will reiterate what 
Jeff H said - it is very unwise to rely on IO forwarding. Much better to just 
directly read the file unless that file is simply unavailable on the node where 
rank=0 is running.

> On Aug 22, 2016, at 1:55 PM, Jingchao Zhang <zh...@unl.edu> wrote:
> 
> Here you can find the source code for lammps input 
> https://github.com/lammps/lammps/blob/r13864/src/input.cpp 
> <https://github.com/lammps/lammps/blob/r13864/src/input.cpp>
> Based on the gdb output, rank 0 stuck at line 167
> if
>  (fgets([m],maxline-m,infile)
>  == NULL)
> and the rest threads stuck at line 203
> MPI_Bcast(,1,MPI_INT,0,world);
> 
> So rank 0 possibly hangs on the fgets() function.
> 
> Here are the whole backtrace information:
> $ cat master.backtrace worker.backtrace
> #0  0x003c37cdb68d in read () from /lib64/libc.so.6
> #1  0x003c37c71ca8 in _IO_new_file_underflow () from /lib64/libc.so.6
> #2  0x003c37c737ae in _IO_default_uflow_internal () from /lib64/libc.so.6
> #3  0x003c37c67e8a in _IO_getline_info_internal () from /lib64/libc.so.6
> #4  0x003c37c66ce9 in fgets () from /lib64/libc.so.6
> #5  0x005c5a43 in LAMMPS_NS::Input::file() () at ../input.cpp:167
> #6  0x005d4236 in main () at ../main.cpp:31
> #0  0x2b1635d2ace2 in poll_dispatch () from 
> /util/opt/openmpi/2.0.0/gcc/6.1.0/lib/libopen-pal.so.20
> #1  0x2b1635d1fa71 in opal_libevent2022_event_base_loop ()
>from /util/opt/openmpi/2.0.0/gcc/6.1.0/lib/libopen-pal.so.20
> #2  0x2b1635ce4634 in opal_progress () from 
> /util/opt/openmpi/2.0.0/gcc/6.1.0/lib/libopen-pal.so.20
> #3  0x2b16351b8fad in ompi_request_default_wait () from 
> /util/opt/openmpi/2.0.0/gcc/6.1.0/lib/libmpi.so.20
> #4  0x2b16351fcb40 in ompi_coll_base_bcast_intra_generic ()
>from /util/opt/openmpi/2.0.0/gcc/6.1.0/lib/libmpi.so.20
> #5  0x2b16351fd0c2 in ompi_coll_base_bcast_intra_binomial ()
>from /util/opt/openmpi/2.0.0/gcc/6.1.0/lib/libmpi.so.20
> #6  0x2b1644fa6d9b in ompi_coll_tuned_bcast_intra_dec_fixed ()
>from /util/opt/openmpi/2.0.0/gcc/6.1.0/lib/openmpi/mca_coll_tuned.so
> #7  0x2b16351cb4fb in PMPI_Bcast () from 
> /util/opt/openmpi/2.0.0/gcc/6.1.0/lib/libmpi.so.20
> #8  0x005c5b5d in LAMMPS_NS::Input::file() () at ../input.cpp:203
> #9  0x005d4236 in main () at ../main.cpp:31
> 
> Thanks,
> 
> Dr. Jingchao Zhang
> Holland Computing Center
> University of Nebraska-Lincoln
> 402-472-6400
> From: users <users-boun...@lists.open-mpi.org 
> <mailto:users-boun...@lists.open-mpi.org>> on behalf of r...@open-mpi.org 
> <mailto:r...@open-mpi.org> <r...@open-mpi.org <mailto:r...@open-mpi.org>>
> Sent: Monday, August 22, 2016 2:17:10 PM
> To: Open MPI Users
> Subject: Re: [OMPI users] stdin issue with openmpi/2.0.0
>  
> Hmmm...perhaps we can break this out a bit? The stdin will be going to your 
> rank=0 proc. It sounds like you have some subsequent step that calls 
> MPI_Bcast?
> 
> Can you first verify that the input is being correctly delivered to rank=0? 
> This will help us isolate if the problem is in the IO forwarding, or in the 
> subsequent Bcast.
> 
>> On Aug 22, 2016, at 1:11 PM, Jingchao Zhang <zh...@unl.edu 
>> <mailto:zh...@unl.edu>> wrote:
>> 
>> Hi all,
>> 
>> We compiled openmpi/2.0.0 with gcc/6.1.0 and intel/13.1.3. Both of them have 
>> odd behaviors when trying to read from standard input.
>> 
>> For example, if we start the application lammps across 4 nodes, each node 16 
>> cores, connected by Intel QDR Infiniband, mpirun works fine for the 1st 
>> time, but always stuck in a few seconds thereafter.
>> Command:
>> mpirun ./lmp_ompi_g++ < in.snr
>> in.snr is the Lammps input file. compiler is gcc/6.1.
>> 
>> Instead, if we use
>> mpirun ./lmp_ompi_g++ -in in.snr
>> it works 100%.
>> 
>> Some odd behaviors we gathered so far. 
>> 1. For 1 node job, stdin always works.
>> 2. For multiple nodes, stdin works unstably when the number of cores per 
>> node are relatively small. For example, for 2/3/4 nodes, each node 8 cores, 
>> mpirun works most of the time. But for each node with >8 cores, mpirun works 
>> the 1st time, then always stuck. There seems to be a magic number when it 
>> stops working.
>> 3. We tested Quantum Expresso with compiler intel/13 and had the same issue. 
>> 
>> We used gdb to debug and found when mpirun was stuck, the rest of the 
>> processes were all waiting on mpi broadcast from the master thread. The 
>> lammps binary, input file and gdb core files 

Re: [OMPI users] stdin issue with openmpi/2.0.0

[r...@open-mpi.org]

Hmmm...perhaps we can break this out a bit? The stdin will be going to your 
rank=0 proc. It sounds like you have some subsequent step that calls MPI_Bcast?

Can you first verify that the input is being correctly delivered to rank=0? 
This will help us isolate if the problem is in the IO forwarding, or in the 
subsequent Bcast.

> On Aug 22, 2016, at 1:11 PM, Jingchao Zhang  wrote:
> 
> Hi all,
> 
> We compiled openmpi/2.0.0 with gcc/6.1.0 and intel/13.1.3. Both of them have 
> odd behaviors when trying to read from standard input.
> 
> For example, if we start the application lammps across 4 nodes, each node 16 
> cores, connected by Intel QDR Infiniband, mpirun works fine for the 1st time, 
> but always stuck in a few seconds thereafter.
> Command:
> mpirun ./lmp_ompi_g++ < in.snr
> in.snr is the Lammps input file. compiler is gcc/6.1.
> 
> Instead, if we use
> mpirun ./lmp_ompi_g++ -in in.snr
> it works 100%.
> 
> Some odd behaviors we gathered so far. 
> 1. For 1 node job, stdin always works.
> 2. For multiple nodes, stdin works unstably when the number of cores per node 
> are relatively small. For example, for 2/3/4 nodes, each node 8 cores, mpirun 
> works most of the time. But for each node with >8 cores, mpirun works the 1st 
> time, then always stuck. There seems to be a magic number when it stops 
> working.
> 3. We tested Quantum Expresso with compiler intel/13 and had the same issue. 
> 
> We used gdb to debug and found when mpirun was stuck, the rest of the 
> processes were all waiting on mpi broadcast from the master thread. The 
> lammps binary, input file and gdb core files (example.tar.bz2) can be 
> downloaded from this link 
> https://drive.google.com/open?id=0B3Yj4QkZpI-dVWZtWmJ3ZXNVRGc 
> 
> 
> Extra information:
> 1. Job scheduler is slurm.
> 2. configure setup:
> ./configure --prefix=$PREFIX \
> --with-hwloc=internal \
> --enable-mpirun-prefix-by-default \
> --with-slurm \
> --with-verbs \
> --with-psm \
> --disable-openib-connectx-xrc \
> --with-knem=/opt/knem-1.1.2.90mlnx1 \
> --with-cma
> 3. openmpi-mca-params.conf file 
> orte_hetero_nodes=1
> hwloc_base_binding_policy=core
> rmaps_base_mapping_policy=core
> opal_cuda_support=0
> btl_openib_use_eager_rdma=0
> btl_openib_max_eager_rdma=0
> btl_openib_flags=1
> 
> Thanks,
> Jingchao 
> 
> Dr. Jingchao Zhang
> Holland Computing Center
> University of Nebraska-Lincoln
> 402-472-6400
> ___
> users mailing list
> users@lists.open-mpi.org 
> https://rfd.newmexicoconsortium.org/mailman/listinfo/users 
> 
___
users mailing list
users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

Re: [OMPI users] Problems with mpirun in openmpi-1.8.1 and -2.0.0

[r...@open-mpi.org]

The rdma error sounds like something isn’t right with your machine’s Infiniband 
installation.

The cross-version problem sounds like you installed both OMPI versions into the 
same location - did you do that?? If so, then that might be the root cause of 
both problems. You need to install them in totally different locations. Then 
you need to _prefix_ your PATH and LD_LIBRARY_PATH with the location of the 
version you want to use.

HTH
Ralph

> On Aug 19, 2016, at 12:53 AM, Juan A. Cordero Varelaq 
>  wrote:
> 
> Dear users,
> 
> I am totally stuck using openmpi. I have two versions on my machine: 1.8.1 
> and 2.0.0, and none of them work. When use the mpirun 1.8.1 version, I get 
> the following error:
> 
> librdmacm: Fatal: unable to open RDMA device
> librdmacm: Fatal: unable to open RDMA device
> librdmacm: Fatal: unable to open RDMA device
> librdmacm: Fatal: unable to open RDMA device
> librdmacm: Fatal: unable to open RDMA device
> --
> Open MPI failed to open the /dev/knem device due to a local error.
> Please check with your system administrator to get the problem fixed,
> or set the btl_sm_use_knem MCA parameter to 0 to run without /dev/knem
> support.
> 
>   Local host: MYMACHINE
>   Errno:  2 (No such file or directory)
> --
> --
> Open MPI failed to open an OpenFabrics device.  This is an unusual
> error; the system reported the OpenFabrics device as being present,
> but then later failed to access it successfully.  This usually
> indicates either a misconfiguration or a failed OpenFabrics hardware
> device.
> 
> All OpenFabrics support has been disabled in this MPI process; your
> job may or may not continue.
> 
>   Hostname:MYMACHINE
>   Device name: mlx4_0
>   Errror (22): Invalid argument
> --
> --
> [[60527,1],4]: A high-performance Open MPI point-to-point messaging module
> was unable to find any relevant network interfaces:
> 
> Module: usNIC
>   Host: MYMACHINE
> 
> When I use the 2.0.0 version, I get something strange, it seems openmpi-2.0.0 
> looks for openmpi-1.8.1 libraries?:
> 
> A requested component was not found, or was unable to be opened.  This
> means that this component is either not installed or is unable to be
> used on your system (e.g., sometimes this means that shared libraries
> that the component requires are unable to be found/loaded).  Note that
> Open MPI stopped checking at the first component that it did not find.
> 
> Host:  MYMACHINE
> Framework: ess
> Component: pmi
> --
> [MYMACHINE:126820] *** Process received signal ***
> [MYMACHINE:126820] Signal: Segmentation fault (11)
> [MYMACHINE:126820] Signal code: Address not mapped (1)
> [MYMACHINE:126820] Failing at address: 0x1c0
> [MYMACHINE:126820] [ 0] 
> /lib/x86_64-linux-gnu/libpthread.so.0(+0xfcb0)[0x7f39b2ec4cb0]
> [MYMACHINE:126820] [ 1] 
> /opt/openmpi-1.8.1/lib/libopen-pal.so.6(opal_libevent2021_event_add+0x10)[0x7f39b23e7430]
> [MYMACHINE:126820] [ 2] 
> /opt/openmpi-1.8.1/lib/libopen-rte.so.7(+0x25a57)[0x7f39b2676a57]
> [MYMACHINE:126820] [ 3] 
> /opt/openmpi-1.8.1/lib/libopen-rte.so.7(orte_show_help_norender+0x197)[0x7f39b2676fb7]
> [MYMACHINE:126820] [ 4] 
> /opt/openmpi-1.8.1/lib/libopen-rte.so.7(orte_show_help+0x10f)[0x7f39b267718f]
> [MYMACHINE:126820] [ 5] 
> /opt/openmpi-1.8.1/lib/libopen-pal.so.6(+0x41f2a)[0x7f39b23c5f2a]
> [MYMACHINE:126820] [ 6] 
> /opt/openmpi-1.8.1/lib/libopen-pal.so.6(mca_base_components_filter+0x273)[0x7f39b23c70c3]
> [MYMACHINE:126820] [ 7] 
> /opt/openmpi-1.8.1/lib/libopen-pal.so.6(mca_base_framework_components_open+0x58)[0x7f39b23c8278]
> [MYMACHINE:126820] [ 8] 
> /opt/openmpi-1.8.1/lib/libopen-pal.so.6(mca_base_framework_open+0x7c)[0x7f39b23d1e6c]
> [MYMACHINE:126820] [ 9] 
> /opt/openmpi-1.8.1/lib/libopen-rte.so.7(orte_init+0x111)[0x7f39b2666e21]
> [MYMACHINE:126820] [10] 
> /opt/openmpi-1.8.1/lib/libmpi.so.1(ompi_mpi_init+0x1c2)[0x7f39b3115c92]
> [MYMACHINE:126820] [11] 
> /opt/openmpi-1.8.1/lib/libmpi.so.1(MPI_Init+0x1ab)[0x7f39b31387bb]
> [MYMACHINE:126820] [12] mb[0x402024]
> [MYMACHINE:126820] [13] 
> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xed)[0x7f39b2b187ed]
> [MYMACHINE:126820] [14] mb[0x402111]
> [MYMACHINE:126820] *** End of error message ***
> --
> A requested component was not found, or was unable to be opened.  This
> means that this component is either not installed or is unable to be
> used on your system (e.g., sometimes this means that shared libraries
> that the component requires are unable to be 

Re: [OMPI users] SGE integration broken in 2.0.0

[r...@open-mpi.org]

> On Aug 12, 2016, at 1:48 PM, Reuti <re...@staff.uni-marburg.de> wrote:
> 
> 
> Am 12.08.2016 um 21:44 schrieb r...@open-mpi.org <mailto:r...@open-mpi.org>:
> 
>> Don’t know about the toolchain issue - I use those same versions, and don’t 
>> have a problem. I’m on CentOS-7, so that might be the difference?
>> 
>> Anyway, I found the missing code to assemble the cmd line for qrsh - not 
>> sure how/why it got deleted.
>> 
>> https://github.com/open-mpi/ompi/pull/1960 
>> <https://github.com/open-mpi/ompi/pull/1960>
> 
> Yep, it's working again - thx.
> 
> But for sure there was a reason behind the removal, which may be elaborated 
> in the Open MPI team to avoid any side effects by fixing this issue.

I actually don’t recall a reason - and I’m the one that generally maintains 
that code area. I think it fell of the map accidentally when I was updating 
that area.

However, we’ll toss it out there for comment - anyone recall?


> 
> -- Reuti
> 
> PS: The other items I'll investigate on Monday.
> 
> 
>>> On Aug 12, 2016, at 12:15 PM, Reuti <re...@staff.uni-marburg.de> wrote:
>>> 
>>>> 
>>>> Am 12.08.2016 um 16:52 schrieb r...@open-mpi.org:
>>>> 
>>>> IIRC, the rationale behind adding the check was that someone using SGE 
>>>> wanted to specify a custom launch agent, and we were overriding it with 
>>>> qrsh. However, the check is incorrect as that MCA param cannot be NULL.
>>>> 
>>>> I have updated this on master - can you see if this fixes the problem for 
>>>> you?
>>>> 
>>>> https://github.com/open-mpi/ompi/pull/1957
>>> 
>>> I updated my tools to:
>>> 
>>> autoconf-2.69
>>> automake-1.15
>>> libtool-2.4.6
>>> 
>>> but I face with Open MPI's ./autogen.pl:
>>> 
>>> configure.ac:152: error: possibly undefined macro: AC_PROG_LIBTOOL
>>> 
>>> I recall seeing in already before, how to get rid of it? For now I fixed 
>>> the single source file just by hand.
>>> 
>>> -- Reuti
>>> 
>>> 
>>>> As for the blank in the cmd line - that is likely due to a space reserved 
>>>> for some entry that you aren’t using (e.g., when someone manually 
>>>> specifies the prefix). It shouldn’t cause any harm as the cmd line parser 
>>>> is required to ignore spaces
>>>> 
>>>> The -ldl problem sounds like a configuration issue - you might want to 
>>>> file a separate issue about it
>>>> 
>>>>> On Aug 11, 2016, at 4:28 AM, Reuti <re...@staff.uni-marburg.de> wrote:
>>>>> 
>>>>> Hi,
>>>>> 
>>>>> In the file orte/mca/plm/rsh/plm_rsh_component I see an if-statement, 
>>>>> which seems to prevent the tight integration with SGE to start:
>>>>> 
>>>>> if (NULL == mca_plm_rsh_component.agent) {
>>>>> 
>>>>> Why is it there (it wasn't in 1.10.3)?
>>>>> 
>>>>> If I just remove it I get:
>>>>> 
>>>>> [node17:25001] [[27678,0],0] plm:rsh: final template argv:
>>>>> qrsh   orted --hnp-topo-sig ...
>>>>> 
>>>>> instead of the former:
>>>>> 
>>>>> /usr/sge/bin/lx24-amd64/qrsh -inherit -nostdin -V -verbose   
>>>>> orted --hnp-topo-sig ...
>>>>> 
>>>>> So, just removing the if-statement is not a perfect cure as the 
>>>>> $SGE_ROOT/$ARC does not prefix `qrsh`.
>>>>> 
>>>>> ==
>>>>> 
>>>>> BTW: why is there blank before " orted" in the assembled command line - 
>>>>> and it's really in the argument when I check this on the slave nodes what 
>>>>> should be started by the `qrsh_starter`? As long as there is a wrapping 
>>>>> shell, it will be removed anyway. But in a special setup we noticed this 
>>>>> additional blank.
>>>>> 
>>>>> ==
>>>>> 
>>>>> I also notice, that I have to supply "-ldl" to `mpicc` to allow the 
>>>>> compilation of an application to succeed in 2.0.0.
>>>>> 
>>>>> -- Reuti
>>>>> ___
>>>>> users mailing list
>>>>> users@lists.open-mpi.org
>>>>> https://rfd.newmexicoconsortium.org/mailman/listinfo/users
>>>> 
>>>> ___
>>>> users mailing list
>>>> users@lists.open-mpi.org
>>>> https://rfd.newmexicoconsortium.org/mailman/listinfo/users
>>>> 
>>> 
>>> ___
>>> users mailing list
>>> users@lists.open-mpi.org
>>> https://rfd.newmexicoconsortium.org/mailman/listinfo/users
>> 
>> ___
>> users mailing list
>> users@lists.open-mpi.org
>> https://rfd.newmexicoconsortium.org/mailman/listinfo/users
> 
> ___
> users mailing list
> users@lists.open-mpi.org <mailto:users@lists.open-mpi.org>
> https://rfd.newmexicoconsortium.org/mailman/listinfo/users 
> <https://rfd.newmexicoconsortium.org/mailman/listinfo/users>
___
users mailing list
users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

Re: [OMPI users] SGE integration broken in 2.0.0

[r...@open-mpi.org]

Don’t know about the toolchain issue - I use those same versions, and don’t 
have a problem. I’m on CentOS-7, so that might be the difference?

Anyway, I found the missing code to assemble the cmd line for qrsh - not sure 
how/why it got deleted.

https://github.com/open-mpi/ompi/pull/1960


> On Aug 12, 2016, at 12:15 PM, Reuti <re...@staff.uni-marburg.de> wrote:
> 
>> 
>> Am 12.08.2016 um 16:52 schrieb r...@open-mpi.org <mailto:r...@open-mpi.org>:
>> 
>> IIRC, the rationale behind adding the check was that someone using SGE 
>> wanted to specify a custom launch agent, and we were overriding it with 
>> qrsh. However, the check is incorrect as that MCA param cannot be NULL.
>> 
>> I have updated this on master - can you see if this fixes the problem for 
>> you?
>> 
>> https://github.com/open-mpi/ompi/pull/1957 
>> <https://github.com/open-mpi/ompi/pull/1957>
> 
> I updated my tools to:
> 
> autoconf-2.69
> automake-1.15
> libtool-2.4.6
> 
> but I face with Open MPI's ./autogen.pl:
> 
> configure.ac:152: error: possibly undefined macro: AC_PROG_LIBTOOL
> 
> I recall seeing in already before, how to get rid of it? For now I fixed the 
> single source file just by hand.
> 
> -- Reuti
> 
> 
>> As for the blank in the cmd line - that is likely due to a space reserved 
>> for some entry that you aren’t using (e.g., when someone manually specifies 
>> the prefix). It shouldn’t cause any harm as the cmd line parser is required 
>> to ignore spaces
>> 
>> The -ldl problem sounds like a configuration issue - you might want to file 
>> a separate issue about it
>> 
>>> On Aug 11, 2016, at 4:28 AM, Reuti <re...@staff.uni-marburg.de> wrote:
>>> 
>>> Hi,
>>> 
>>> In the file orte/mca/plm/rsh/plm_rsh_component I see an if-statement, which 
>>> seems to prevent the tight integration with SGE to start:
>>> 
>>>  if (NULL == mca_plm_rsh_component.agent) {
>>> 
>>> Why is it there (it wasn't in 1.10.3)?
>>> 
>>> If I just remove it I get:
>>> 
>>> [node17:25001] [[27678,0],0] plm:rsh: final template argv:
>>>  qrsh   orted --hnp-topo-sig ...
>>> 
>>> instead of the former:
>>> 
>>> /usr/sge/bin/lx24-amd64/qrsh -inherit -nostdin -V -verbose   
>>> orted --hnp-topo-sig ...
>>> 
>>> So, just removing the if-statement is not a perfect cure as the 
>>> $SGE_ROOT/$ARC does not prefix `qrsh`.
>>> 
>>> ==
>>> 
>>> BTW: why is there blank before " orted" in the assembled command line - and 
>>> it's really in the argument when I check this on the slave nodes what 
>>> should be started by the `qrsh_starter`? As long as there is a wrapping 
>>> shell, it will be removed anyway. But in a special setup we noticed this 
>>> additional blank.
>>> 
>>> ==
>>> 
>>> I also notice, that I have to supply "-ldl" to `mpicc` to allow the 
>>> compilation of an application to succeed in 2.0.0.
>>> 
>>> -- Reuti
>>> ___
>>> users mailing list
>>> users@lists.open-mpi.org
>>> https://rfd.newmexicoconsortium.org/mailman/listinfo/users
>> 
>> ___
>> users mailing list
>> users@lists.open-mpi.org
>> https://rfd.newmexicoconsortium.org/mailman/listinfo/users
>> 
> 
> ___
> users mailing list
> users@lists.open-mpi.org <mailto:users@lists.open-mpi.org>
> https://rfd.newmexicoconsortium.org/mailman/listinfo/users 
> <https://rfd.newmexicoconsortium.org/mailman/listinfo/users>
___
users mailing list
users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

Re: [OMPI users] mpirun won't find programs from the PATH environment variable that are in directories that are relative paths

[r...@open-mpi.org]

Sorry for the delay - I had to catchup on some other things before I could come 
back to checking this one. Took me awhile to track this down, but the change is 
in test for master:

https://github.com/open-mpi/ompi/pull/1958

Once complete, I’ll set it up for inclusion in v2.0.1

Thanks for reporting it!
Ralph


> On Jul 29, 2016, at 5:47 PM, Phil Regier  wrote:
> 
> If I'm reading you right, you're presently unable to do the equivalent 
> (albeit probably with PATH set on a different line somewhere above) of
> 
> PATH=arch/x86_64-rhel7-gcc48-opt/bin mpirun -n 1 psana
> 
> I'm mildly curious whether it would help to add a leading "./" to get the 
> equivalent of
> 
> PATH=./arch/x86_64-rhel7-gcc48-opt/bin mpirun -n 1 psana
> 
> But to be clear, I'm advocating
> 
> PATH=$PWD/arch/x86_64-rhel7-gcc48-opt/bin mpirun -n 1 psana
> 
> as opposed to
> 
> mpirun -n 1 $PWD/arch/x86_64-rhel7-gcc48-opt/bin/psana
> 
> mostly because you still get to set the path once and use it many times 
> without duplicating code.
> 
> 
> For what it's worth, I've seen Ralph's suggestion generalized to something 
> like
> 
> PREFIX=$PWD/arch/x86_64-rhel7-gcc48-opt/bin mpirun -n 1 $PREFIX/psana
> 
> where PREFIX might be set above in the same script, or sourced from a common 
> config script or a custom environment module.  I think this style appeals to 
> many users on many levels.
> 
> 
> In any event, though, if this really is a bug that gets fixed, you've got 
> lots of options.
> 
> 
> 
> 
> On Fri, Jul 29, 2016 at 5:24 PM, Schneider, David A. 
> > wrote:
> Hi, Thanks for the reply! It does look like mpirun runs from the same 
> directory as where I launch it, and that the environment has the same value 
> for PATH that I had before (with the relative directory in front), but of 
> course, there are lots of other MPI based environment variables defined - 
> maybe one of those means don't use the relative paths?
> 
> Explicitly setting the path with $PWD like you say, yes, I agree that is a 
> good defensive practice, but it is more cumbersome, the actually path looks
> 
>  mpirun -n 1 $PWD/arch/x86_64-rhel7-gcc48-opt/bin/psana
> 
> best,
> 
> David Schneider
> SLAC/LCLS
> 
> From: users [users-boun...@lists.open-mpi.org 
> ] on behalf of Phil Regier 
> [preg...@penguincomputing.com ]
> Sent: Friday, July 29, 2016 5:12 PM
> To: Open MPI Users
> Subject: Re: [OMPI users] mpirun won't find programs from the PATH 
> environment variable that are in directories that are relative paths
> 
> I might be three steps behind you here, but does "mpirun  pwd" show 
> that all your launched processes are running in the same directory as the 
> mpirun command?  I assume that "mpirun  env" would show that your PATH 
> variable is being passed along correctly, since you don't have any problems 
> with absolute paths.  In any event, is PATH=$PWD/dir/bin not an option?
> 
> Seems to me that this last would be good practice for location-sensitive 
> launches in general, though I do tend to miss things.
> 
> On Fri, Jul 29, 2016 at 4:34 PM, Schneider, David A. 
>   >> wrote:
> I am finding, on linux, rhel7, with openmpi 1.8.8 and 1.10.3, that mpirun 
> won't find apps that are specified on a relative path, i.e, if I have
> 
> PATH=dir/bin
> 
> and I am in a directory which has dir/bin as a subdirectory, and an 
> executable bir/bin/myprogram, I can't do
> 
> mpirun myprogram
> 
> I get the error message that
> 
> mpirun was unable to find the specified executable file, and therefore
> did not launch the job.
> 
> whereas if I put an absolute path, something like
> 
> PATH=/home/me/dir/bin
> 
> then it works.
> 
> This causes some problematic silent failure, sometimes we use relative 
> directories to override a 'base' release, so if I had
> 
> PATH=dir/bin:/central/install/dir/bin
> 
> and myprogram was in both dir/bin and /central/install/dir/bin, through 
> mpirun, I would be running myprogram from the central install, but otherwise 
> I would run it from my own directory.
> 
> Do other people find this is the case? I wonder if it is a problem that got 
> introduced through our installation of openmpi.  We do create relocatable 
> rpm's, and I'm also trying openmpi from a conda package that is relocatable, 
> I think all the prefix paths in the binary and text files were corrected 
> properly for the install - at least everything else seems to work.
> 
> best,
> 
> David Schneider
> SLAC/LCLS
> ___
> users mailing list
> users@lists.open-mpi.org 
>  >
> 

Re: [OMPI users] SGE integration broken in 2.0.0

[r...@open-mpi.org]

IIRC, the rationale behind adding the check was that someone using SGE wanted 
to specify a custom launch agent, and we were overriding it with qrsh. However, 
the check is incorrect as that MCA param cannot be NULL.

I have updated this on master - can you see if this fixes the problem for you?

https://github.com/open-mpi/ompi/pull/1957

As for the blank in the cmd line - that is likely due to a space reserved for 
some entry that you aren’t using (e.g., when someone manually specifies the 
prefix). It shouldn’t cause any harm as the cmd line parser is required to 
ignore spaces

The -ldl problem sounds like a configuration issue - you might want to file a 
separate issue about it

> On Aug 11, 2016, at 4:28 AM, Reuti  wrote:
> 
> Hi,
> 
> In the file orte/mca/plm/rsh/plm_rsh_component I see an if-statement, which 
> seems to prevent the tight integration with SGE to start:
> 
>if (NULL == mca_plm_rsh_component.agent) {
> 
> Why is it there (it wasn't in 1.10.3)?
> 
> If I just remove it I get:
> 
> [node17:25001] [[27678,0],0] plm:rsh: final template argv:
>qrsh   orted --hnp-topo-sig ...
> 
> instead of the former:
> 
> /usr/sge/bin/lx24-amd64/qrsh -inherit -nostdin -V -verbose   orted 
> --hnp-topo-sig ...
> 
> So, just removing the if-statement is not a perfect cure as the 
> $SGE_ROOT/$ARC does not prefix `qrsh`.
> 
> ==
> 
> BTW: why is there blank before " orted" in the assembled command line - and 
> it's really in the argument when I check this on the slave nodes what should 
> be started by the `qrsh_starter`? As long as there is a wrapping shell, it 
> will be removed anyway. But in a special setup we noticed this additional 
> blank.
> 
> ==
> 
> I also notice, that I have to supply "-ldl" to `mpicc` to allow the 
> compilation of an application to succeed in 2.0.0.
> 
> -- Reuti
> ___
> users mailing list
> users@lists.open-mpi.org
> https://rfd.newmexicoconsortium.org/mailman/listinfo/users

___
users mailing list
users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

Re: [OMPI users] OPENSHMEM ERROR with 2+ Distributed Machines

[r...@open-mpi.org]

Just as a suggestion: most of us are leery of opening Word attachments on 
mailing lists. I’d suggest sending this to us as plain text if you want us to 
read it.


> On Aug 12, 2016, at 4:03 AM, Debendra Das  wrote:
> 
> I have installed OpenMPI-2.0.0 in 5 systems with IP addresses 172.16.5.29, 
> 172.16.5.30, 172.16.5.31, 172.16.5.32, 172.16.5.33.While executing the 
> hello_oshmem_c.c program (under the examples directory) , correct output is 
> coming only when executing is done using 2 distributed machines.But error is 
> coming when 3 or more distributed machines are used.The outputs and the host 
> file  are attached.Can anybody please help me to sort out this error?
> 
> Thanking You.
> Debendranath Das 
> ___
> users mailing list
> users@lists.open-mpi.org
> https://rfd.newmexicoconsortium.org/mailman/listinfo/users

___
users mailing list
users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/users