Re: [OMPI devel] [TIPC BTL] test programmes

[Chris Samuel]

On Mon, 1 Aug 2011 09:47:00 PM Xin He wrote:

> Do any of you guys have any testing programs that I should
> run to test if it really works?

How about a real MPI program which has test data to check
it's running OK ?  Gromacs is open source and has a self-test
mechanism run via "make test" IIRC.

I think HPL (Linpack) also checks the data from its run..

cheers,
Chris
-- 
   Christopher Samuel - Senior Systems Administrator
 VLSCI - Victorian Life Sciences Computation Initiative
 Email: sam...@unimelb.edu.au Phone: +61 (0)3 903 55545
 http://www.vlsci.unimelb.edu.au/

Re: [OMPI devel] [torquedev] Communication between Torque and MPI

[Chris Samuel]

On Thu, 25 Aug 2011 09:07:48 PM Jayavant Patil wrote:

> Hi,

Hiya,

> Is anybody having a tutorial or reference pages
> explaining about the communication between Torque
> and MPI?

Open-MPI uses the PBS Task Manager (TM) API to talk to
Torque pbs_mom's.  If you have the Torque manual pages
installed you can "man tm" to see the docs.

The code in Open-MPI that launches jobs is in the code:

$openmpi/orte/mca/plm/tm/

Hope that helps!
Chris
-- 
   Christopher Samuel - Senior Systems Administrator
 VLSCI - Victorian Life Sciences Computation Initiative
 Email: sam...@unimelb.edu.au Phone: +61 (0)3 903 55545
 http://www.vlsci.unimelb.edu.au/

Re: [OMPI devel] btl/openib: get_ib_dev_distance doesn't see processes as bound if the job has been launched by srun

[Chris Samuel]

On Thursday 09 February 2012 22:18:20 Jeff Squyres wrote:

> Just so that I understand this better -- if a process is bound in a
> cpuset, will tools like hwloc's lstopo only show the Linux
> processors *in that cpuset*?  I.e., does it not have any
> visibility of the processors outside of its cpuset?

I believe that was the intention - there's no real benefit to knowing 
about resources that you can't access or use (and will likely get an 
error if you do) to my mind.

cheers!
Chris
-- 
   Christopher Samuel - Senior Systems Administrator
 VLSCI - Victorian Life Sciences Computation Initiative
 Email: sam...@unimelb.edu.au Phone: +61 (0)3 903 55545
 http://www.vlsci.unimelb.edu.au/

[OMPI devel] help-mpi-btl-openib.txt needs updating with real btl_openib_ib_min_rnr_timer and btl_openib_ib_timeout defaults

[Chris Samuel]

Hi all,

We've been working trying to track down an IB issue here where a
user was having code (Gromacs, run with OMPI 1.4.5) dieing with:

[[18115,1],2][btl_openib_component.c:3224:handle_wc] from bruce030 to: bruce130 
error polling LP CQ with status 
RETRY EXCEEDED ERROR status number 12 for wr_id 7406080 opcode 0 vendor error 
129 qp_idx 2

The odd thing I've spotted though is that in the error it says:

* btl_openib_ib_retry_count - The number of times the sender will attempt to 
retry (defaulted to 7, the maximum 
value).
* btl_openib_ib_timeout - The local ACK timeout parameter (defaulted to 10).

Those don't match the values compiled into OMPI 1.4.5:

ompi_info -a | egrep 'btl_openib_ib_min_rnr_timer|btl_openib_ib_timeout'
 MCA btl: parameter "btl_openib_ib_min_rnr_timer" (current value: "25",
data source: default value)
 MCA btl: parameter "btl_openib_ib_timeout" (current value: "20", data
source: default value)

It looks like the file:

 ompi/mca/btl/openib/help-mpi-btl-openib.txt

needs to be updated with the correct values.

We're stuck on 1.4 for the forseable future (too many apps to
recompile) so I don't know if 1.5+ has the same issue.

It's been there since at least 2009.. :-)

http://www.open-mpi.org/community/lists/users/2009/03/8600.php

cheers!
Chris
-- 
   Christopher Samuel - Senior Systems Administrator
 VLSCI - Victorian Life Sciences Computation Initiative
 Email: sam...@unimelb.edu.au Phone: +61 (0)3 903 55545
 http://www.vlsci.unimelb.edu.au/

Re: [OMPI devel] help-mpi-btl-openib.txt needs updating with real btl_openib_ib_min_rnr_timer and btl_openib_ib_timeout defaults

[Chris Samuel]

On Tuesday 13 March 2012 10:06:43 Chris Samuel wrote:

> Those don't match the values compiled into OMPI 1.4.5:
> 
> ompi_info -a | egrep
> 'btl_openib_ib_min_rnr_timer|btl_openib_ib_timeout' MCA btl:
> parameter "btl_openib_ib_min_rnr_timer" (current value: "25", data
> source: default value)
>  MCA btl: parameter "btl_openib_ib_timeout" (current value: "20",
> data source: default value)

Wrong command line, sigh..

# ompi_info -a | egrep 'MCA.*(btl_openib_ib_retry_count|btl_openib_ib_timeout)'
 MCA btl: parameter "btl_openib_ib_timeout" (current value: 
"20", data source: default value)
 MCA btl: parameter "btl_openib_ib_retry_count" (current value: 
"7", data source: default value)

Even more oddly, the second run of the users job did print
that 7 and 20 as the defaults.

I suspect this is likely the result of the user accidentally
using an earlier version of OMPI, not 1.4.5, for his first run.

Sorry for the noise!

cheers,
Chris
-- 
   Christopher Samuel - Senior Systems Administrator
 VLSCI - Victorian Life Sciences Computation Initiative
 Email: sam...@unimelb.edu.au Phone: +61 (0)3 903 55545
 http://www.vlsci.unimelb.edu.au/

[OMPI devel] Possible OMPI 1.6.5 bug? SEGV in malloc.c

[Chris Samuel]

Hi folks,

One of our users (oh, OK, our director, one of the Dalton developers)
has found an odd behaviour of OMPI 1.6.5 on our x86 clusters and has
managed to get a small reproducer - a modified version of the
ubiquitous F90 "hello world" MPI program.

We find that if we run this program (compiled with either Intel or GCC)
after doing "ulimit -v $((1*1024*1024))" to simulate the default 1GB
memory limit for jobs under Slurm we get odd, but different behaviour.

With the Intel compilers it appears to just hang, but if I run it under
strace I see it looping constantly SEGV'ing.

With RHEL 6.4 gfortran it instead SEGV's straight away and gives a
stack trace:

 Hello, world, I am0  of1
[barcoo:27489] *** Process received signal ***
[barcoo:27489] Signal: Segmentation fault (11)
[barcoo:27489] Signal code: Address not mapped (1)
[barcoo:27489] Failing at address: 0x2008e5708
[barcoo:27489] [ 0] /lib64/libpthread.so.0() [0x3f7b60f500]
[barcoo:27489] [ 1] 
/usr/local/openmpi/1.6.5/lib/libmpi.so.1(opal_memory_ptmalloc2_int_malloc+0x982)
 [0x7f83caff6dd2]
[barcoo:27489] [ 2] 
/usr/local/openmpi/1.6.5/lib/libmpi.so.1(opal_memory_ptmalloc2_malloc+0x52) 
[0x7f83caff7f42]
[barcoo:27489] [ 3] ./gnumyhello_f90(MAIN__+0x146) [0x400f6a]
[barcoo:27489] [ 4] ./gnumyhello_f90(main+0x2a) [0x4011ea]
[barcoo:27489] [ 5] /lib64/libc.so.6(__libc_start_main+0xfd) [0x3f7b21ecdd]
[barcoo:27489] [ 6] ./gnumyhello_f90() [0x400d69]
[barcoo:27489] *** End of error message ***


If I let it generate a core file "bt" tells me:

(gdb) bt
#0  sYSMALLOc (av=0xefd0, bytes=) at 
malloc.c:3240
#1  opal_memory_ptmalloc2_int_malloc (av=0xefd0, bytes=) at malloc.c:4328
#2  0x7f83caff7f42 in opal_memory_ptmalloc2_malloc (bytes=856000) at 
malloc.c:3433
#3  0x00400f6a in main () at gnumyhello_f90.f90:26
#4  0x004011ea in main ()


I've attached his reproducer program, I've just compiled it with:

mpif90 -g -o ./gnumyhello_f90 gnumyhello_f90.f90

We've reproduced it on two different Intel clusters (both RHEL 6.4,
one Nehalem and one SandyBridge) so I'd be really interested to
know if this is a bug?

Thanks!
Chris
-- 
 Christopher SamuelSenior Systems Administrator
 VLSCI - Victorian Life Sciences Computation Initiative
 Email: sam...@unimelb.edu.au Phone: +61 (0)3 903 55545
 http://www.vlsci.org.au/  http://twitter.com/vlsci
!
! Copyright (c) 2004-2006 The Trustees of Indiana University and Indiana
! University Research and Technology
! Corporation.  All rights reserved.
! Copyright (c) 2004-2005 The Regents of the University of California.
! All rights reserved.
! Copyright (c) 2006  Cisco Systems, Inc.  All rights reserved.
!
! Sample MPI "hello world" application in Fortran 90
!
program main
use mpi
implicit none
integer :: ierr, rank, size

!integer, parameter :: WRKMEM=1050*10**6
integer, parameter :: WRKMEM=1070*10**6

real (kind(0.d0)), allocatable, dimension(:) :: work

call MPI_INIT(ierr)
call MPI_COMM_RANK(MPI_COMM_WORLD, rank, ierr)
call MPI_COMM_SIZE(MPI_COMM_WORLD, size, ierr)
print *, "Hello, world, I am ", rank, " of ", size

allocate(work(WRKMEM),stat=ierr)
if (ierr .eq. 0) then
   print *, "Task ", rank, " successfully allocated ", &
(8.d0*WRKMEM/(1024**3)), "GB"
   deallocate(work)
else
   print *, "Task ", rank, " failed to allocate ", &
(8.d0*WRKMEM/(1024**3)), "GB"
end if

call MPI_FINALIZE(ierr)
end

Re: [OMPI devel] SC13 birds of a feather

[Chris Samuel]

On Wed, 4 Dec 2013 11:39:29 AM Jeff Squyres wrote:

> On Dec 3, 2013, at 7:54 PM,
> Christopher Samuel  wrote:
>
> > Would it make any sense to expose system/environmental/thermal
> > information to the application via MPI_T ?
> 
> Hmm.  Interesting idea.

Phew. :-)

> Is the best way to grab such stuff via IPMI?

I don't think so, that means either having the process have permissions to 
access /dev/ipmi* or needing to talk over the network to the adapter, neither 
of which are likely to be desirable (or even possible, our iDataplex IMMs are 
not accessible from the compute nodes).

However, using the coretemp kernel module means you get access to at least 
information about CPU temperatures on Intel systems:

/sys/bus/platform/devices/coretemp.${A}/temp${B}_input

which contains the core temperature in 100ths of a degree Celsius and are 
world readable.  You also get access to the various thermal trip points and 
alarms.

The ${B} value is 1 for the CPU package (SandyBridge or later only), then 
sequentially for the physical cores.  ${A} is 0 for the first socket, then 
max($B of $A)+1 for the next socket, etc..

So on the test login node of our 2010 era Nehalem iDataplex you get a file per 
CPU core but nothing for the socket, viz:

[root@merri-test ~]# ls /sys/bus/platform/devices/coretemp.*/*input*
/sys/bus/platform/devices/coretemp.0/temp2_input
/sys/bus/platform/devices/coretemp.0/temp3_input
/sys/bus/platform/devices/coretemp.0/temp4_input
/sys/bus/platform/devices/coretemp.0/temp5_input
/sys/bus/platform/devices/coretemp.4/temp2_input
/sys/bus/platform/devices/coretemp.4/temp3_input
/sys/bus/platform/devices/coretemp.4/temp4_input
/sys/bus/platform/devices/coretemp.4/temp5_input

[root@merri-test ~]# cat /sys/bus/platform/devices/coretemp.*/*input*
52000
52000
52000
53000
59000
55000
58000
56000

On the test login node of our SandyBridge iDataplex delivered mid year we get 
the package as well:

[root@barcoo-test ~]# ls /sys/bus/platform/devices/coretemp.*/*input*
/sys/bus/platform/devices/coretemp.0/temp1_input
/sys/bus/platform/devices/coretemp.0/temp2_input
/sys/bus/platform/devices/coretemp.0/temp3_input
/sys/bus/platform/devices/coretemp.0/temp4_input
/sys/bus/platform/devices/coretemp.0/temp5_input
/sys/bus/platform/devices/coretemp.0/temp6_input
/sys/bus/platform/devices/coretemp.0/temp7_input
/sys/bus/platform/devices/coretemp.6/temp1_input
/sys/bus/platform/devices/coretemp.6/temp2_input
/sys/bus/platform/devices/coretemp.6/temp3_input
/sys/bus/platform/devices/coretemp.6/temp4_input
/sys/bus/platform/devices/coretemp.6/temp5_input
/sys/bus/platform/devices/coretemp.6/temp6_input
/sys/bus/platform/devices/coretemp.6/temp7_input

[root@barcoo-test ~]# cat /sys/bus/platform/devices/coretemp.*/*input*
44000
43000
44000
42000
43000
38000
44000
37000
33000
37000
32000
34000
36000
33000

There's more information in $KERNEL_SOURCE/Documentation/hwmon/coretemp.

Both those systems are running RHEL6, so it should be fairly well supported 
*if* the sysadmin has loaded the modules.

> That might well be do-able, since there's no performance penalty for reading
> such values until you actually read the values (i.e., we don't actively
> monitor these values in OMPI's overall progression engine; they're only
> read when the application invokes an MPI_T read function).

Indeed, these *shouldn't* hang trying to read them. ;-)

cheers,
Chris
-- 
 Christopher SamuelSenior Systems Administrator
 VLSCI - Victorian Life Sciences Computation Initiative
 Email: sam...@unimelb.edu.au Phone: +61 (0)3 903 55545
 http://www.vlsci.org.au/  http://twitter.com/vlsci

Re: [OMPI devel] RFC: Force Slurm to use PMI-1 unless PMI-2 is specifically requested

[Chris Samuel]

On Thu, 8 May 2014 09:10:00 AM Joshua Ladd wrote:

> We (MLNX) are working on a new SLURM PMI2 plugin that we plan to eventually
> push upstream. However, to use it, it will require linking in a proprietary
> Mellanox library that accelerates the collective operations (available in
> MOFED versions 2.1 and higher.)

What about those of us who cannot run Mellanox OFED?

All the best,
Chris
-- 
 Christopher SamuelSenior Systems Administrator
 VLSCI - Victorian Life Sciences Computation Initiative
 Email: sam...@unimelb.edu.au Phone: +61 (0)3 903 55545
 http://www.vlsci.org.au/  http://twitter.com/vlsci

[OMPI devel] OpenMPI, PLPA and Linux cpuset/cgroup support

[Chris Samuel]

Hi all,

Not sure if this is a OpenMPI query or a PLPA query,
but given that PLPA seems to have some support for it
already I thought I'd start here. :-)

We run a quad core Opteron cluster with Torque 2.3.x
which uses the kernels cpuset support to constrain
a job to just the cores it has been allocated.

However, we are seeing occasionally that where a job
has been allocated multiple cores on the same node we
get two compute bound MPI processes in the job scheduled
onto the same core (obviously a kernel issue).

So CPU affinity would be an obvious solution, but it
needs to be done with reference to the cores that are
available to it in its cpuset.

This information is already retrievable by PLPA (for
instance "plpa-taskset -cp $$" will retrieve the cores
allocated to the shell you run the command from) but
I'm not sure if OpenMPI makes use of this when binding
CPUs using the linux paffinity MCA parameter ?

Our testing (with 1.3.2) seems to show it doesn't, and
I don't think there are any significant differences with
the snapshots in 1.4.

Am I correct in this ?  If so, are there any plans to
make it do this ?

cheers,
Chris
-- 
Christopher Samuel - (03) 9925 4751 - Systems Manager
 The Victorian Partnership for Advanced Computing
 P.O. Box 201, Carlton South, VIC 3053, Australia
VPAC is a not-for-profit Registered Research Agency

Re: [OMPI devel] OpenMPI, PLPA and Linux cpuset/cgroup support

[Chris Samuel]

- "Ralph Castain"  wrote:

Hi Ralph,

> Interesting. No, we don't take PLPA cpu sets into account when  
> retrieving the allocation.

Understood.

> Just to be clear: from an OMPI perspective, I don't think this is an 
> issue of binding, but rather an issue of allocation. If we knew we had
> been allocated only a certain number of cores on a node, then we would
> only map that many procs to the node. When we subsequently "bind", we 
> should then bind those procs to the correct cores (I think).

Hmm, OpenMPI should already know this from the PBS TM API when
launching the job, we've never had to get our users to specify
how many procs per node to start (and they will generally have
no idea how many to ask for in advance as they are at the mercy
of the scheduler, unless they select a whole nodes with ppn=8).

> Could you check this? You can run a trivial job using the
> -npernode x option, where x matched the #cores you were
> allocated on the nodes.
>
> If you do this, do we bind to the correct cores?

I'll give this a shot tomorrow when I'm back in the office
(just checking email late at night here), I'll try it under
strace to to see what it tries to sched_setaffinity() to.

> If we do, then that would confirm that we just aren't
> picking up the right number of cores allocated to us.
> If it is wrong, then this is a PLPA issue where it
> isn't binding to the right core.

Interesting, will let you know the test results tomorrow!

cheers,
Chris
-- 
Christopher Samuel - (03) 9925 4751 - Systems Manager
 The Victorian Partnership for Advanced Computing
 P.O. Box 201, Carlton South, VIC 3053, Australia
VPAC is a not-for-profit Registered Research Agency

Re: [OMPI devel] OpenMPI, PLPA and Linux cpuset/cgroup support

[Chris Samuel]

- "Ralph Castain"  wrote:

> Could you check this? You can run a trivial job using the -npernode x 
> option, where x matched the #cores you were allocated on the nodes.
> If you do this, do we bind to the correct cores?

Nope, I'm afraid it doesn't - submitted a job asking
for 4 cores on one node and was allocated cores 0-3 in
the cpuset.

Grep'ing the strace output for anything mentioning affinity shows:

[csamuel@tango027 CPI]$ fgrep affinity cpi-trace.txt
11412 execve("/usr/local/openmpi/1.3.3-gcc/bin/mpiexec", ["mpiexec", "--mca", 
"paffinity", "linux", "-npernode", "4", "/home/csamuel/Sources/Tests/CPI/"...], 
[/* 56 vars */]) = 0
11412 sched_getaffinity(0, 128,  { f }) = 8
11412 sched_setaffinity(0, 8,  { 0 })   = -1 EFAULT (Bad address)
11416 sched_getaffinity(0, 128,  
11416 <... sched_getaffinity resumed>  { f }) = 8
11416 sched_setaffinity(0, 8,  { 0 } 
11416 <... sched_setaffinity resumed> ) = -1 EFAULT (Bad address)
11414 sched_getaffinity(0, 128,  
11414 <... sched_getaffinity resumed>  { f }) = 8
11414 sched_setaffinity(0, 8,  { 0 } 
11414 <... sched_setaffinity resumed> ) = -1 EFAULT (Bad address)
11413 sched_getaffinity(0, 128,  
11413 <... sched_getaffinity resumed>  { f }) = 8
11413 sched_setaffinity(0, 8,  { 0 } 
11413 <... sched_setaffinity resumed> ) = -1 EFAULT (Bad address)
11415 sched_getaffinity(0, 128,  
11415 <... sched_getaffinity resumed>  { f }) = 8
11415 sched_setaffinity(0, 8,  { 0 } 
11415 <... sched_setaffinity resumed> ) = -1 EFAULT (Bad address)
11413 sched_getaffinity(11413, 8,  
11415 sched_getaffinity(11415, 8,  
11413 <... sched_getaffinity resumed>  { f }) = 8
11415 <... sched_getaffinity resumed>  { f }) = 8
11414 sched_getaffinity(11414, 8,  
11414 <... sched_getaffinity resumed>  { f }) = 8
11416 sched_getaffinity(11416, 8,  
11416 <... sched_getaffinity resumed>  { f }) = 8

I can confirm that it's not worked by checking what
plpa-taskset says about a process (for example 11414):

[root@tango027 plpa-taskset]# ./plpa-taskset -cp 11414
pid 11414's current affinity list: 0-3

According to the manual page:

   EFAULT A supplied memory address was invalid.

This is on a dual socket quad core AMD Shanghai system
running the 2.6.28.9 kernel (not had a chance to upgrade
recently).

Will do some more poking around after lunch.

cheers,
Chris
-- 
Christopher Samuel - (03) 9925 4751 - Systems Manager
 The Victorian Partnership for Advanced Computing
 P.O. Box 201, Carlton South, VIC 3053, Australia
VPAC is a not-for-profit Registered Research Agency

Re: [OMPI devel] OpenMPI, PLPA and Linux cpuset/cgroup support

[Chris Samuel]

- "Ralph Castain"  wrote:

> Looking at your command line, did you remember to set -mca  
> mpi_paffinity_alone 1?

Ahh, no, sorry, still feeling my way with this..

> If not, we won't set affinity on the processes.

Now it fails immediately with:

  Setting processor affinity failed
  --> Returned "Invalid argument" (-11) instead of "Success" (0)

wrapped in a bunch of OpenMPI messages explaining that it
failed on start.

The strace looks much the same as before.

[csamuel@tango047 CPI]$ fgrep affinity cpi-trace.txt
10853 execve("/usr/local/openmpi/1.3.3-gcc/bin/mpiexec", ["mpiexec", "-mca", 
"mpi_paffinity_alone", "1", "-npernode", "4", 
"/home/csamuel/Sources/Tests/CPI/"...], [/* 56 vars */]) = 0
10853 sched_getaffinity(0, 128,  { 3c }) = 8
10853 sched_setaffinity(0, 8,  { 0 })   = -1 EFAULT (Bad address)
10854 sched_getaffinity(0, 128,  
10854 <... sched_getaffinity resumed>  { 3c }) = 8
10854 sched_setaffinity(0, 8,  { 0 } 
10854 <... sched_setaffinity resumed> ) = -1 EFAULT (Bad address)
10857 sched_getaffinity(0, 128,  
10857 <... sched_getaffinity resumed>  { 3c }) = 8
10857 sched_setaffinity(0, 8,  { 0 } 
10857 <... sched_setaffinity resumed> ) = -1 EFAULT (Bad address)
10856 sched_getaffinity(0, 128,  
10856 <... sched_getaffinity resumed>  { 3c }) = 8
10856 sched_setaffinity(0, 8,  { 0 } 
10856 <... sched_setaffinity resumed> ) = -1 EFAULT (Bad address)
10855 sched_getaffinity(0, 128,  
10855 <... sched_getaffinity resumed>  { 3c }) = 8
10855 sched_setaffinity(0, 8,  { 0 } 
10855 <... sched_setaffinity resumed> ) = -1 EFAULT (Bad address)
10857 sched_setaffinity(10857, 8,  { 8 } 
10857 <... sched_setaffinity resumed> ) = 0
10856 sched_setaffinity(10856, 8,  { 4 } 
10856 <... sched_setaffinity resumed> ) = 0
10854 sched_setaffinity(10854, 8,  { 1 } 
10854 <... sched_setaffinity resumed> ) = -1 EINVAL (Invalid argument)


cheers,
Chris
-- 
Christopher Samuel - (03) 9925 4751 - Systems Manager
 The Victorian Partnership for Advanced Computing
 P.O. Box 201, Carlton South, VIC 3053, Australia
VPAC is a not-for-profit Registered Research Agency

Re: [OMPI devel] OpenMPI, PLPA and Linux cpuset/cgroup support

[Chris Samuel]

- "Ralph Castain"  wrote:

> Sounds like a problem in PLPA - I'll have to defer
> to them.

Understood, thanks for that update.  I'll try and
find some time to look inside PLPA too. 

> Our primary PLPA person is on vacation this week, so
> you might not hear back from him until later next week
> when he gets through his inbox mountain.

I can quite sympathise!  I'm away on leave next week
so it might be a little time until we can resynchronise. :-)

> PLPA may have its own mailing list too - not really sure.

Just seen the link from Terry (thanks!), will take a
look at how busy it is.

All the best,
Chris
-- 
Christopher Samuel - (03) 9925 4751 - Systems Manager
 The Victorian Partnership for Advanced Computing
 P.O. Box 201, Carlton South, VIC 3053, Australia
VPAC is a not-for-profit Registered Research Agency

Re: [OMPI devel] OpenMPI, PLPA and Linux cpuset/cgroup support

[Chris Samuel]

- "Ralph Castain"  wrote:

> Looking at your command line, did you remember to set -mca  
> mpi_paffinity_alone 1? If not, we won't set affinity on the
> processes.

Just realised that in the failed test I posted I set
-mca mpi_affinity_alone 1 *instead* of -mca paffinity linux,
rather than as well as.

Would that result in PLPA not being used ?

Is there a debugging switch to get mpiexec to show what
MCA modules are being loaded/used at runtime ?

cheers,
Chris
-- 
Christopher Samuel - (03) 9925 4751 - Systems Manager
 The Victorian Partnership for Advanced Computing
 P.O. Box 201, Carlton South, VIC 3053, Australia
VPAC is a not-for-profit Registered Research Agency

Re: [OMPI devel] OpenMPI, PLPA and Linux cpuset/cgroup support

[Chris Samuel]

- "Ralph Castain"  wrote:

> This will tell you what module is loaded. If PLPA -can- run, you  
> should see the linux module selected.

Thanks Ralph, yes it is being selected.  I'll carry
on digging.

cheers!
Chris
-- 
Christopher Samuel - (03) 9925 4751 - Systems Manager
 The Victorian Partnership for Advanced Computing
 P.O. Box 201, Carlton South, VIC 3053, Australia
VPAC is a not-for-profit Registered Research Agency

Re: [OMPI devel] OpenMPI, PLPA and Linux cpuset/cgroup support

[Chris Samuel]

- "Chris Samuel"  wrote:

> I'll carry on digging.

I've been trying to track back from the linux
paffinity module to find some useful debugging
info I can get my teeth into and I can see that
the file:

 opal/mca/paffinity/base/paffinity_base_service.c 

seems to have lots of useful debugging message that
should output what processes are getting bound where,
but they are often conditional on the diag_requested
variable.  It seems that diag_requested is set based
on this logic:

if (4 < opal_output_get_verbosity(opal_paffinity_base_output)) {
diag_requested = true;
} else {
diag_requested = false;
}

But I can't see what I have to set to make those messages
appear! I've been through the output of ompi_info -a in case
there was something listed there but nothing leapt out at me.

Any clues please ?

cheers,
Chris
-- 
Christopher Samuel - (03) 9925 4751 - Systems Manager
 The Victorian Partnership for Advanced Computing
 P.O. Box 201, Carlton South, VIC 3053, Australia
VPAC is a not-for-profit Registered Research Agency

Re: [OMPI devel] OpenMPI, PLPA and Linux cpuset/cgroup support

[Chris Samuel]

- "Ralph Castain"  wrote:

> Should just be
> 
> -mca paffinity_base_verbose 5
> 
> Any value greater than 4 should turn it "on"

Yup, that's what I was trying, but couldn't get any output.

> Something I should have mentioned. The paffinity_base_service.c  file 
> is solely used by the rank_file syntax. It has nothing to do with  
> setting mpi_paffinity_alone and letting OMPI self-determine the  
> process-to-core binding.

That would explain why I'm not seeing any output from it
then, it and the solaris module are the only ones containing
any opal_output() statements in the paffinity section of MCA.

I'll try scattering some opal_output()'s into the linux module
instead along the same lines as the base module.

> You want to dig into the linux module code that calls down
> into the plpa. The same mca param should give you messages
> from the module, and -might- give you messages from inside
> plpa (not sure of the latter).

The PLPA output is not run time selectable:

#if defined(PLPA_DEBUG) && PLPA_DEBUG && 0

:-)

cheers,
Chris
-- 
Christopher Samuel - (03) 9925 4751 - Systems Manager
 The Victorian Partnership for Advanced Computing
 P.O. Box 201, Carlton South, VIC 3053, Australia
VPAC is a not-for-profit Registered Research Agency

Re: [OMPI devel] OpenMPI, PLPA and Linux cpuset/cgroup support

[Chris Samuel]

- "Jeff Squyres"  wrote:

Hi Jeff,

> I'm the "primary PLPA" guy that Ralph referred to, and I was on  
> vacation last week -- sorry for missing all the chatter.

No worries!

> Based on your mails, it looks like you're out this week -- so little 
> will likely occur.  I'm at the MPI Forum standards meeting next week, 
> so my replies to email will be sporatic.

Not a problem, can quite understand.

> OMPI is pretty much directly calling PLPA to set affinity for  
> "processors" 0, 1, 2, 3 (which PLPA translates into Linux virtual  
> processor IDs, and then invokes sched_setaffinity with each of those 
> IDs).

Cool, so it does indeed sound like something that can be
solved purely inside PLPA.  That's good to know!

> Note that the EFAULT errors you're seeing in the output are  
> deliberate. [...]

Great, after reading a bit more I got the impression that's
what might be going on, thanks for the confirmation!

> But as to why it's getting EINVAL, that could be wonky.
> We might want to take this to the PLPA list and have you
> run some small, non-MPI examples to ensure that PLPA is
> parsing your /sys tree properly, etc.

Not a problem.

> Ping when you get back from vacation.

I'm back Monday (which is Sunday arvo for you I think).

cheers!
Chris
-- 
Christopher Samuel - (03) 9925 4751 - Systems Manager
 The Victorian Partnership for Advanced Computing
 P.O. Box 201, Carlton South, VIC 3053, Australia
VPAC is a not-for-profit Registered Research Agency

Re: [OMPI devel] OpenMPI, PLPA and Linux cpuset/cgroup support

[Chris Samuel]

Hi Bert,

- "Bert Wesarg"  wrote:

> The Cpus_allowed* fields in /proc//status are the same as
> sched_getaffinity returns and the /proc//cpuset needs to be
> resolved, i.e. where is the cpuset fs mounted?

The convention is to mount it on /dev/cpuset.

Unfortunately you cannot mount both the cpuset and
the cgroup virtual filesystems concurrently and with
the more advanced features of control groups such as
device whitelisting, memory usage limits, etc (and the
fact that cpusets is now a subset of cgroups) you will
need to be able to cope with either of these being in use.

Certainly it's something I'm concerned with as I'm
thinking about refactoring the Torque cpuset support
to be more generic.

cheers,
Chris
-- 
Christopher Samuel - (03) 9925 4751 - Systems Manager
 The Victorian Partnership for Advanced Computing
 P.O. Box 201, Carlton South, VIC 3053, Australia
VPAC is a not-for-profit Registered Research Agency

Re: [OMPI devel] OpenMPI, PLPA and Linux cpuset/cgroup support

[Chris Samuel]

- "Jeff Squyres"  wrote:

> PLPA does not currently deal with cpusets.

I think it can get close enough if it assumes that
its initial affinity list is the subset of cores that
it can choose from when setting CPU affinity.

As for whether OMPI or PLPA should choose, I suspect
it's better if OMPI continues to be dumb and say "bind
me to CPU n" and PLPA works out what CPU 'n' works
out to be in terms of actual cores it can access.

But I'll take the rest of this to the PLPA list. :-)

cheers,
Chris
-- 
Christopher Samuel - (03) 9925 4751 - Systems Manager
 The Victorian Partnership for Advanced Computing
 P.O. Box 201, Carlton South, VIC 3053, Australia
VPAC is a not-for-profit Registered Research Agency

Re: [OMPI devel] OpenMPI, PLPA and Linux cpuset/cgroup support

[Chris Samuel]

Hi Ralph,

- "Ralph Castain"  wrote:

> UmmmI'll let you guys work this out on PLPA. However, just to  
> clarify, OMPI currently binds to cores, not logical cpus. It is the  
> PLPA that is "dumb" and provides the plumbing to do what OMPI tells
> it.
> 
> :-)

Ahh, if that's the case then this isn't something that PLPA
needs to be involved in then!  I'm getting confused here,
haven't had a coffee since Monday, must be the reason. :-)

What we really want is for the usual execution of:

mpiexec /path/to/my/code $arguments

to launch via the PBS TM API, work out what cores are
available to it on the nodes assigned to it and then
bind appropriately to them.  Easy, see.. ;-)

cheers!
Chris
-- 
Christopher Samuel - (03) 9925 4751 - Systems Manager
 The Victorian Partnership for Advanced Computing
 P.O. Box 201, Carlton South, VIC 3053, Australia
VPAC is a not-for-profit Registered Research Agency

Re: [OMPI devel] OpenMPI, PLPA and Linux cpuset/cgroup support

[Chris Samuel]

- "Ralph Castain"  wrote:

> Perhaps a telecon (myself, Jeff S, and you) would be best at this  
> stage.

Sounds good, will take that part to private email.

> I confess I'm now confused too - what you describe is precisely
> what we already do.

I added printf()'s to the PLPA init(),
PLPA_NAME(sched_getaffinity)() and
PLPA_NAME(sched_setaffinity)() functions to see where
they are getting called to try and clarify what's up.

I do see init() and PLPA_NAME(sched_getaffinity)() getting
called, but never PLPA_NAME(sched_getaffinity)().

This is on my home box (quad core) not our production
clusters, but the basics should be the same..

cheers,
Chris
-- 
Christopher Samuel - (03) 9925 4751 - Systems Manager
 The Victorian Partnership for Advanced Computing
 P.O. Box 201, Carlton South, VIC 3053, Australia
VPAC is a not-for-profit Registered Research Agency

Re: [OMPI devel] Shared library versioning

[Chris Samuel]

- "Ralf Wildenhues"  wrote:

> You can probably solve most of these issues by just
> versioning the directory names where you put the files;

To be honest I'm not sure if this is something that
OMPI should be looking to solve, we have lots of
different versions installed on our clusters just
using --prefix to set a different install location.

We manage all those using Modules (modules.sf.net)
where we have a .base module which works out from its
name where the installation directory would be, and
then new configurations are just a symlink away.

cheers,
Chris
-- 
Christopher Samuel - (03) 9925 4751 - Systems Manager
 The Victorian Partnership for Advanced Computing
 P.O. Box 201, Carlton South, VIC 3053, Australia
VPAC is a not-for-profit Registered Research Agency

Re: [OMPI devel] Heads up on new feature to 1.3.4

[Chris Samuel]

- "Terry Dontje"  wrote:

> I just wanted to give everyone a heads up if they do not get bugs 
> email.  I just submitted a CMR to move over some new paffinity options
> from the trunk to the v1.3 branch.

Ralphs comments imply that for those sites that share nodes
between jobs (such as ourselves, and most other sites that
I'm aware of in Australia) these changes will severely impact
performance.

I think that would be a Very Bad Thing(tm).

Can it be something that defaults to being configured out
for at least 1.3 please ?  That way those few sites that
can take advantage can enable it whilst the rest of us
aren't impacted.

cheers,
Chris
-- 
Christopher Samuel - (03) 9925 4751 - Systems Manager
 The Victorian Partnership for Advanced Computing
 P.O. Box 201, Carlton South, VIC 3053, Australia
VPAC is a not-for-profit Registered Research Agency

Re: [OMPI devel] Heads up on new feature to 1.3.4

[Chris Samuel]

- "Ralph Castain"  wrote:

> Hi Chris

Hiya Ralph,

> There would be a "-do-not-bind" option that will prevent us from
> binding processes to anything which should cover that situation.

Gotcha.

> My point was only that we would be changing the out-of-the-box
> behavior to the opposite of today's, so all those such as yourself
> would now have to add the -do-not-bind MCA param to your default MCA
> param file.
>
> Doable - but it -is- a significant change in our out-of-the-box
> behavior.

I think this is too big a change in the default behaviour
for a stable release, it'll cause a lot of people pain for
no readily apparent reason.

I also believe that if those sites with multiple MPI jobs
on nodes are indeed in the majority then it makes more sense
to keep the default behaviour and have those who need this
functionality enable it on their installs.

Thoughts ?

cheers,
Chris
-- 
Christopher Samuel - (03) 9925 4751 - Systems Manager
 The Victorian Partnership for Advanced Computing
 P.O. Box 201, Carlton South, VIC 3053, Australia
VPAC is a not-for-profit Registered Research Agency

Re: [OMPI devel] Heads up on new feature to 1.3.4

[Chris Samuel]

- "Eugene Loh"  wrote:

> This is an important discussion.

Indeed! My big fear is that people won't pick up the significance
of the change and will complain about performance regressions
in the middle of an OMPI stable release cycle.

> Do note:
> 
> 1) Bind-to-core is actually the default behavior of many MPIs today.

We had this issue with MVAPICH before we dumped it to go to OpenMPI
as if we had (for example) two 4 core jobs running on the same node
they'd both go at half speed whilst the node itself was 50% idle.

Turned out they'd both bound to cores 0-3 leaving cores 4-7 unused. :-(

Fortunately there was an undocumented environment variable
that let us turn it off for all jobs, but getting rid of that
misbehaviour was a major reason for switching to OpenMPI.

> 2) The proposed OMPI bind-to-socket default is less severe. In the
> general case, it would allow multiple jobs to bind in the same way
> without oversubscribing any core or socket. (This comment added to
> the trac ticket.)

That's a nice clarification, thanks. I suspect though that the
same issue we have with MVAPICH would occur if two 4 core jobs
both bound themselves to the first socket.

Thinking further, it would be interesting to find out how this
code would behave on a system where cpusets is in use and so OMPI
has to submit to the will of the scheduler regarding cores/sockets.

> 3) Defaults (if I understand correctly) can be set differently
> on each cluster.

Yes, but the defaults should be sensible for the majority of
clusters.  If the majority do indeed share nodes between jobs
then I would suggest that the default should be off and the
minority who don't share nodes should have to enable it.

There's also the issue of those users who (for whatever reason)
like to build their own MPI stack and who are even less likely
to understand the impact that they may have on others.. :-(

cheers!
Chris
-- 
Christopher Samuel - (03) 9925 4751 - Systems Manager
 The Victorian Partnership for Advanced Computing
 P.O. Box 201, Carlton South, VIC 3053, Australia
VPAC is a not-for-profit Registered Research Agency

Re: [OMPI devel] Heads up on new feature to 1.3.4

[Chris Samuel]

- "Jeff Squyres"  wrote:

> An important point to raise here: the 1.3 series is *not* the super  
> stable series.  It is the *feature* series.  Specifically: it is not 
> out of scope to introduce or change features within the 1.3 series.

Ah, I think I've misunderstood the website then. :-(

It calls 1.3 stable and 1.2 old and I presumed old
meant deprecated. :-(

-- 
Christopher Samuel - (03) 9925 4751 - Systems Manager
 The Victorian Partnership for Advanced Computing
 P.O. Box 201, Carlton South, VIC 3053, Australia
VPAC is a not-for-profit Registered Research Agency

Re: [OMPI devel] Heads up on new feature to 1.3.4

[Chris Samuel]

- "Eugene Loh"  wrote:

Hi Eugene,

[...]
> It would be even better to have binding selections adapt to other
> bindings on the system.

Indeed!

This touches on the earlier thread about making OMPI aware
of its cpuset/cgroup allocation on the node (for those sites
that are using it), it might solve this issue quite nicely as
OMPI would know precisely what cores & sockets were allocated
for its use without having to worry about other HPC processes.

No idea how to figure that out for processes outside of cpusets. :-(

> In any case, regardless of what the best behavior is, I appreciate
> the point about changing behavior in the middle of a stable release.

Not a problem, and I take Jeff's point about 1.3 not being a
super stable release and thus not being a blocker to changes
such as this.

> Arguably, leaving significant performance on the table in typical
> situations is a bug that warrants fixing even in the middle of a
> release, but I won't try to settle that debate here.

I agree for those cases where there's no downside, and thinking
further on your point of balancing between sockets I can see why
that would limit the impact.

Most of the cases I can think of that would be most adversely
affected are down to other jobs binding to cores naively and if
that's happening outside of cpusets then the cluster sysadmin
has more to worry about from mixing those applications than
mixing with OMPI ones which are just binding to sockets. :-)

So I'll happily withdraw my objection on those grounds.

*But* I would like to test this code out on a cluster with
cpuset support enabled to see whether it will behave itself.

Basically if I run a 4 core MPI job on a dual socket system
which has been allocated only the cores on socket 0 what will
happen when it tries to bind to socket 1 which is outside its
cpuset ?

Is there a 1.3 branch or tarball with these patches applied
that I could test out ?

cheers,
Chris
-- 
Christopher Samuel - (03) 9925 4751 - Systems Manager
 The Victorian Partnership for Advanced Computing
 P.O. Box 201, Carlton South, VIC 3053, Australia
VPAC is a not-for-profit Registered Research Agency

Re: [OMPI devel] Heads up on new feature to 1.3.4

[Chris Samuel]

- "Ralph Castain"  wrote:

> Hi Chris

Hiya,

> The devel trunk has all of this in it - you can get that tarball from 
> the OMPI web site (take the nightly snapshot).

OK, grabbed that (1.4a1r21825). Configured with:

./configure --prefix=$FOO --with-openib --with-tm=/usr/
local/torque/latest --enable-static  --enable-shared

It built & installed OK, but when running a trivial example
with it I don't see evidence for that code getting called.
Perhaps I'm not passing the correct options ?

$ mpiexec -bysocket -bind-to-socket -mca odls_base_report_bindings 99 -mca 
odls_base_verbose 7 ./cpi-1.4
[tango095.vpac.org:16976] mca:base:select:( odls) Querying component [default]
[tango095.vpac.org:16976] mca:base:select:( odls) Query of component [default] 
set priority to 1
[tango095.vpac.org:16976] mca:base:select:( odls) Selected component [default]
[tango095.vpac.org:16976] [[36578,0],0] odls:launch: spawning child 
[[36578,1],0]
[tango095.vpac.org:16976] [[36578,0],0] odls:launch: spawning child 
[[36578,1],1]
[tango095.vpac.org:16976] [[36578,0],0] odls:launch: spawning child 
[[36578,1],2]
[tango095.vpac.org:16976] [[36578,0],0] odls:launch: spawning child 
[[36578,1],3]
Process 0 on tango095.vpac.org
Process 1 on tango095.vpac.org
Process 2 on tango095.vpac.org
Process 3 on tango095.vpac.org
^Cmpiexec: killing job...

Increasing odls_base_verbose only seems to add the environment being
passed to the child processes. :-(

I'm pretty sure I've got the right code as ompi_info -a
reports the debug setting from the patch:

MCA odls: parameter "odls_base_report_bindings" (current value: <0>, data 
source: default value)

> I plan to work on cpuset support beginning Tues morning.

Great, anything I can help with then please let me know,
I'll be back from leave by then.

All the best,
Chris
-- 
Christopher Samuel - (03) 9925 4751 - Systems Manager
 The Victorian Partnership for Advanced Computing
 P.O. Box 201, Carlton South, VIC 3053, Australia
VPAC is a not-for-profit Registered Research Agency

Re: [OMPI devel] Heads up on new feature to 1.3.4

[Chris Samuel]

- "Eugene Loh"  wrote:

> Ah, you're missing the third secret safety switch that prevents
> hapless mortals from using this stuff accidentally!  :^)

Sounds good to me. :-)

> I think you need to add
> 
> --mca opal_paffinity_alone 1


Yup, looks like that's it; it fails to launch with that..


$ mpiexec --mca opal_paffinity_alone 1 -bysocket -bind-to-socket -mca 
odls_base_report_bindings 99 -mca odls_base_verbose 7 ./cpi-1.4
[tango095.vpac.org:18548] mca:base:select:( odls) Querying component [default]
[tango095.vpac.org:18548] mca:base:select:( odls) Query of component [default] 
set priority to 1
[tango095.vpac.org:18548] mca:base:select:( odls) Selected component [default]
[tango095.vpac.org:18548] [[33990,0],0] odls:launch: spawning child 
[[33990,1],0]
[tango095.vpac.org:18548] [[33990,0],0] odls:launch: spawning child 
[[33990,1],1]
[tango095.vpac.org:18548] [[33990,0],0] odls:default:fork binding child 
[[33990,1],0] to socket 0 cpus 000f
[tango095.vpac.org:18548] [[33990,0],0] odls:default:fork binding child 
[[33990,1],1] to socket 1 cpus 00f0
--
An attempt to set processor affinity has failed - please check to
ensure that your system supports such functionality. If so, then
this is probably something that should be reported to the OMPI developers.
--
--
mpiexec was unable to start the specified application as it encountered an error
on node tango095.vpac.org. More information may be available above.
--
4 total processes failed to start


This is most likely because it's getting an error from the
kernel when trying to bind to a socket it's not permitted
to access.

cheers,
Chris
-- 
Christopher Samuel - (03) 9925 4751 - Systems Manager
 The Victorian Partnership for Advanced Computing
 P.O. Box 201, Carlton South, VIC 3053, Australia
VPAC is a not-for-profit Registered Research Agency

Re: [OMPI devel] Heads up on new feature to 1.3.4

[Chris Samuel]

- "Chris Samuel"  wrote:

> This is most likely because it's getting an error from the
> kernel when trying to bind to a socket it's not permitted
> to access.

This is what strace reports:

18561 sched_setaffinity(18561, 8,  { f0 } 
18561 <... sched_setaffinity resumed> ) = -1 EINVAL (Invalid argument)

so that would appear to be it.

cheers,
Chris
-- 
Christopher Samuel - (03) 9925 4751 - Systems Manager
 The Victorian Partnership for Advanced Computing
 P.O. Box 201, Carlton South, VIC 3053, Australia
VPAC is a not-for-profit Registered Research Agency

Re: [OMPI devel] Heads up on new feature to 1.3.4

[Chris Samuel]

- "Chris Samuel"  wrote:

> $ mpiexec --mca opal_paffinity_alone 1 -bysocket -bind-to-socket -mca
> odls_base_report_bindings 99 -mca odls_base_verbose 7 ./cpi-1.4

To clarify - does that command line accurately reflect the
proposed defaults for OMPI 1.3.4 ?

cheers,
Chris
-- 
Christopher Samuel - (03) 9925 4751 - Systems Manager
 The Victorian Partnership for Advanced Computing
 P.O. Box 201, Carlton South, VIC 3053, Australia
VPAC is a not-for-profit Registered Research Agency

Re: [OMPI devel] Heads up on new feature to 1.3.4

[Chris Samuel]

- "Eugene Loh"  wrote:

> Actually, the current proposed defaults for 1.3.4 are
> not to change the defaults at all.

Thanks, I hadn't picked up on the latest update to the
trac ticket 3 days ago that says that the defaults will
stay the same. Sounds good to me!

All the best and have a good weekend all!
Chris
-- 
Christopher Samuel - (03) 9925 4751 - Systems Manager
 The Victorian Partnership for Advanced Computing
 P.O. Box 201, Carlton South, VIC 3053, Australia
VPAC is a not-for-profit Registered Research Agency

Re: [OMPI devel] RFC: convert send to ssend

[Chris Samuel]

- "Jeff Squyres"  wrote:

> Does anyone have any suggestions?  Or are we stuck
> with compile-time checking?

I didn't see this until now, but I'd be happy with
just a compile time option so we could produce an
install just for debugging purposes and have our
users explicitly select it with modules.

I have to say that this is of interest to us as we're
trying to help a researcher at one of our member uni's
to track down a bug where a message appears to go missing.

cheers!
Chris
-- 
Christopher Samuel - (03) 9925 4751 - Systems Manager
 The Victorian Partnership for Advanced Computing
 P.O. Box 201, Carlton South, VIC 3053, Australia
VPAC is a not-for-profit Registered Research Agency

Re: [OMPI devel] RFC: convert send to ssend

[Chris Samuel]

- "George Bosilca"  wrote:

> Do people know that there exist tools for checking MPI code  
> correctness? Many, many tools and most of them are freely
> available.

Yes, but have yet to be able to persuade any of our users
to use them (and have no control over them). :-(

-- 
Christopher Samuel - (03) 9925 4751 - Systems Manager
 The Victorian Partnership for Advanced Computing
 P.O. Box 201, Carlton South, VIC 3053, Australia
VPAC is a not-for-profit Registered Research Agency

Re: [OMPI devel] XML request

[Chris Samuel]

- "Ralph Castain"  wrote:

> Hi Greg
> 
> I fixed these so they will get properly formatted. However, it is
> symptomatic of a much broader problem - namely, that developers have
> inserted print statements throughout the code for reporting errors.
> There simply isn't any easy way for me to catch them all.

Would something like Coccinelle which lets you do semantic
patching across an entire codebase be useful ?

 http://coccinelle.lip6.fr/

There's a nice rundown of its use with the Linux kernel
by Val Aurora, the Linux filesystem developer, on LWN here:

 http://lwn.net/Articles/315686/

That's from January, so there have been fixes since then
(last release was at the end of July).

cheers,
Chris
-- 
Christopher Samuel - (03) 9925 4751 - Systems Manager
 The Victorian Partnership for Advanced Computing
 P.O. Box 201, Carlton South, VIC 3053, Australia
VPAC is a not-for-profit Registered Research Agency

Re: [OMPI devel] suffix flag problems

[Chris Samuel]

- "David Robertson"  wrote:

> Hi all,

Hiya,

> We use both the PGI and Intel compilers over an
> Infiniband cluster and I was trying to find a way
> to have both orteruns in the path (in separate
> directories) at the same time.

Not a solution, but what we do here is to arrange our
installs into separate locations thus:

/usr/local/openmpi-gcc/$version
/usr/local/openmpi-intel/$version
/usr/local/openmpi-pgi/$version

and then use Modules (modules.sf.net) to let users easily
select which they want to use, so to load the default OMPI
for the Intel compilers they do:

module load openmpi-intel

We use symlinks for the module files to a generic one which
uses some lrange and split in tcl to derive the application
and compiler from the name of the module and loads the
compiler module if it's not already been loaded.

Users load a "vpac" module from their profile/cshrc at
login time and that pulls in our suggested defaults for
that cluster.

Makes life a lot easier for our users (and us!).

cheers,
Chris
-- 
Christopher Samuel - (03) 9925 4751 - Systems Manager
 The Victorian Partnership for Advanced Computing
 P.O. Box 201, Carlton South, VIC 3053, Australia
VPAC is a not-for-profit Registered Research Agency

Re: [OMPI devel] RFC - "system-wide-only" MCA parameters

[Chris Samuel]

- "Ralph Castain"  wrote:

> Let me point out the obvious since this has plagued
> us at LANL with regard to this concept. If a user
> wants to do something different, all they have to
> do is download and build their own copy of OMPI.

One possibility may be to have OMPI honour a config
file in /etc by default as well as the ones in $prefix,
so if a user built their own copy then they would have
to hack the code in order to disable the check.

I also really liked the idea of OMPI outputing a
message if a user tried to override a locked param,
much better to explain why something is bad rather
than just saying "no" (especially if you can say
"no, but you could try...").

cheers!
Chris
-- 
Christopher Samuel - (03) 9925 4751 - Systems Manager
 The Victorian Partnership for Advanced Computing
 P.O. Box 201, Carlton South, VIC 3053, Australia
VPAC is a not-for-profit Registered Research Agency

Re: [OMPI devel] application hangs with multiple dup

[Chris Samuel]

Hi Edgar,

- "Edgar Gabriel"  wrote:

> just wanted to give a heads-up that I *think* I know what the problem
> is. I should have a fix (with a description) either later today or 
> tomorrow morning...

I see that changeset 21970 is on trunk to fix this issue,
is that backportable to the 1.3.x branch ?

Love to see if this fixes up our users issues with Gadget!

cheers,
Chris
-- 
Christopher Samuel - (03) 9925 4751 - Systems Manager
 The Victorian Partnership for Advanced Computing
 P.O. Box 201, Carlton South, VIC 3053, Australia
VPAC is a not-for-profit Registered Research Agency

Re: [OMPI devel] application hangs with multiple dup

[Chris Samuel]

Hi Edgar,

- "Edgar Gabriel"  wrote:

> it will be available in 1.3.4...

That's great, thanks so much!

cheers,
Chris
-- 
Christopher Samuel - (03) 9925 4751 - Systems Manager
 The Victorian Partnership for Advanced Computing
 P.O. Box 201, Carlton South, VIC 3053, Australia
VPAC is a not-for-profit Registered Research Agency

Re: [OMPI devel] application hangs with multiple dup

[Chris Samuel]

Hi Terry,

- "Terry Dontje"  wrote:

> It's actually is in the 1.3 branch now and has been
> verified to solve the hanging issues of several members.

Great, I'll get them to try a snapshot build!

cheers,
Chris
-- 
Christopher Samuel - (03) 9925 4751 - Systems Manager
 The Victorian Partnership for Advanced Computing
 P.O. Box 201, Carlton South, VIC 3053, Australia
VPAC is a not-for-profit Registered Research Agency

[OMPI devel] OMPI 1.3.4 ETA ? (TLAs FTW)

[Chris Samuel]

Hi folks,

Just wondered if there was any idea of when OMPI 1.3.4
might be released ?  I know the correct answer is "when
it's ready" (:-)) but was curious if there was any thoughts
on a timeframe ?

The cpuset aware CPU affinity code would be very useful
to us to fix up some codes that sometimes get stuck sharing
cores with others free (presumably a kernel process scheduler
bug)..

cheers!
Chris
-- 
Christopher Samuel - (03) 9925 4751 - Systems Manager
 The Victorian Partnership for Advanced Computing
 P.O. Box 201, Carlton South, VIC 3053, Australia
VPAC is a not-for-profit Registered Research Agency

Re: [OMPI devel] MPI_Graph_create

[Chris Samuel]

- "David Singleton"  wrote:

> Kiril Dichev has already pointed a problem with MPI_Cart_create
> http://www.open-mpi.org/community/lists/devel/2009/08/6627.php
> MPI_Graph_create has the same problem.  I checked all other
> functions with logical in arguments and no others do anything
> similar.

I suspect the issue (looking at trunk) is at line 113
of ompi/mpi/c/graph_create.c :

re_order = (1 == reorder) ? true:false;

Which means that re_order is only true if 1 is passed
through via the reorder parameter.

I suspect changing it to:

re_order = (0 == reorder) ? false:true;

would fix it as then re_order is only false if 0 is
passed through (assuming you define 0 as false and
!0 as true).  Tested via a little harness program
and it seems to do the right thing.

cheers,
Chris
-- 
Christopher Samuel - (03) 9925 4751 - Systems Manager
 The Victorian Partnership for Advanced Computing
 P.O. Box 201, Carlton South, VIC 3053, Australia
VPAC is a not-for-profit Registered Research Agency

Re: [OMPI devel] 0.9.1rc2 is available

[Chris Samuel]

- "Jeff Squyres"  wrote:

> Give it a whirl:

Nice - built without warnings with GCC 4.4.2.

Some sample results below for configs not represented
on the current website.


Dual socket Shanghai:

System(31GB)
  Node#0(15GB) + Socket#0 + L3(6144KB)
L2(512KB) + L1(64KB) + Core#0 + P#0
L2(512KB) + L1(64KB) + Core#1 + P#1
L2(512KB) + L1(64KB) + Core#2 + P#2
L2(512KB) + L1(64KB) + Core#3 + P#3
  Node#1(16GB) + Socket#1 + L3(6144KB)
L2(512KB) + L1(64KB) + Core#0 + P#4
L2(512KB) + L1(64KB) + Core#1 + P#5
L2(512KB) + L1(64KB) + Core#2 + P#6
L2(512KB) + L1(64KB) + Core#3 + P#7


Dual socket single core Opteron:

System(3961MB)
  Node#0(2014MB) + Socket#0 + L2(1024KB) + L1(1024KB) + Core#0 + P#0
  Node#1(2017MB) + Socket#1 + L2(1024KB) + L1(1024KB) + Core#0 + P#1


Dual socket, dual core Power5 (SMT disabled) running SLES9
(2.6.9 based kernel):

System(15GB)
  Node#0(7744MB)
P#0
P#2
  Node#1(8000MB)
P#4
P#6


Inside a single CPU Torque job (using cpusets) on a dual socket Shanghai:

System(31GB)
  Node#0(15GB) + Socket#0 + L3(6144KB) + L2(512KB) + L1(64KB) + Core#0 + P#0
  Node#1(16GB)


-- 
Christopher Samuel - (03) 9925 4751 - Systems Manager
 The Victorian Partnership for Advanced Computing
 P.O. Box 201, Carlton South, VIC 3053, Australia
VPAC is a not-for-profit Registered Research Agency

Re: [OMPI devel] 0.9.1rc2 is available

[Chris Samuel]

- "Chris Samuel"  wrote:

> Some sample results below for configs not represented
> on the current website.

A final example of a more convoluted configuration with
a Torque job requesting 5 CPUs on a dual Shanghai node
and has been given a non-contiguous configuration.

[csamuel@tango069 ~]$ cat /dev/cpuset/`cat /proc/$$/cpuset`/cpus
0,4-7

[csamuel@tango069 ~]$ ~/local/hwloc/0.9.1rc2/bin/lstopo
System(31GB)
  Node#0(15GB) + Socket#0 + L3(6144KB) + L2(512KB) + L1(64KB) + Core#0 + P#0
  Node#1(16GB) + Socket#1 + L3(6144KB)
L2(512KB) + L1(64KB) + Core#0 + P#4
L2(512KB) + L1(64KB) + Core#1 + P#5
L2(512KB) + L1(64KB) + Core#2 + P#6
L2(512KB) + L1(64KB) + Core#3 + P#7

-- 
Christopher Samuel - (03) 9925 4751 - Systems Manager
 The Victorian Partnership for Advanced Computing
 P.O. Box 201, Carlton South, VIC 3053, Australia
VPAC is a not-for-profit Registered Research Agency

Re: [OMPI devel] 0.9.1rc2 is available

[Chris Samuel]

- "Jeff Squyres"  wrote:

> Sweet!

:-)

> And -- your reply tells me that, for the 2nd time in a single day, I 
> posted to the wrong list.  :-)

Ah well, if you'd posted to the right list I wouldn't
have seen this.

> I'll forward your replies to the hwloc-devel list.

Not a problem - I'll go subscribe now.

cheers!
Chris
-- 
Christopher Samuel - (03) 9925 4751 - Systems Manager
 The Victorian Partnership for Advanced Computing
 P.O. Box 201, Carlton South, VIC 3053, Australia
VPAC is a not-for-profit Registered Research Agency

Re: [OMPI devel] 0.9.1rc2 is available

[Chris Samuel]

- "Tony Breeds"  wrote:

> Powerpc kernels that old do not have the topology information needed
> (in /sys or /proc/cpuinfo)  So for the short term that's be best we
> can do. 

That's fine, I quite understand.  I'm trying to get that
cluster replaced anyway.. ;-)

> FWIW I'm looking at how we can pull more (if not all) the
> same info from the device tree on these kenels.

Interesting idea, best of luck with that!

All the best,
Chris
-- 
Christopher Samuel - (03) 9925 4751 - Systems Manager
 The Victorian Partnership for Advanced Computing
 P.O. Box 201, Carlton South, VIC 3053, Australia
VPAC is a not-for-profit Registered Research Agency

Re: [OMPI devel] 0.9.1rc2 is available

[Chris Samuel]

- "Ashley Pittman"  wrote:

> $ grep Cpus_allowed_list /proc/$$/status

Useful, ta!

> Does this imply the default is to report on processes
> in the current cpuset rather than the entire system?
> Does anyone else feel that violates the principal of
> least surprise?

Not really, I feel that when you are in a cpuset that
*is* the entire system as far as you are concerned.

cheers,
Chris
-- 
Christopher Samuel - (03) 9925 4751 - Systems Manager
 The Victorian Partnership for Advanced Computing
 P.O. Box 201, Carlton South, VIC 3053, Australia
VPAC is a not-for-profit Registered Research Agency

Re: [OMPI devel] RFC: Use automake "silent rules"

[Chris Samuel]

- "Jeff Squyres"  wrote:

> WHAT: Change OMPI's verbose build output to use
> Automake's new "silent" rules output (see below)

Not that I'm a developer, but as someone who regularly
builds OMPI I'd be very happy with this change.

cheers!
Chris
-- 
Christopher Samuel - (03) 9925 4751 - Systems Manager
 The Victorian Partnership for Advanced Computing
 P.O. Box 201, Carlton South, VIC 3053, Australia
VPAC is a not-for-profit Registered Research Agency

Re: [OMPI devel] MPI_Graph_create

[Chris Samuel]

- "Jeff Squyres"  wrote:

> You are absolutely correct.  I've filed CMRs
> for v1.4 and v1.5.

To clarify one point for people who weren't at the
SC'09 Open-MPI BOF (hopefully I'll get this right!):

1.4 will be the bug-fix only continuation of the 1.3
feature series and will be binary compatible with 1.3.x
where x >= 2.

cheers,
Chris
-- 
Christopher Samuel - (03) 9925 4751 - Systems Manager
 The Victorian Partnership for Advanced Computing
 P.O. Box 201, Carlton South, VIC 3053, Australia
VPAC is a not-for-profit Registered Research Agency

Re: [OMPI devel] MALLOC_MMAP_MAX (and MALLOC_MMAP_THRESHOLD)

[Chris Samuel]

On Sun, 10 Jan 2010 03:55:33 am Eugene Loh wrote:

> The default (I think this means Linux, whether with gcc, gfortran, Sun f90,
> etc.) is to use mmap to malloc large allocations.

This was brought in by glibc version 2.3:

 http://en.wikipedia.org/wiki/Malloc#dlmalloc_.28the_glibc_allocator.29

It has the side effect of making the behaviour of malloc() in response to 
resource limits dependent on the size of the allocation being made. For 
example RLIMIT_DATA (maximum size of a processes data segment) will only be 
enforced for small allocations that don't trigger the use of mmap().

Had great fun with that trying to track down why the mem property of Torque 
PBS jobs wasn't being enforced all the time.

cheers,
Chris
-- 
 Chris Samuel  :  http://www.csamuel.org/  :  Melbourne, VIC

This email may come with a PGP signature as a file. Do not panic.
For more info see: http://en.wikipedia.org/wiki/OpenPGP


signature.asc
Description: This is a digitally signed message part.

Re: [OMPI devel] RFC: Adding -DOPEN_MPI=1 to mpif77 and mpif90

[Chris Samuel]

Jeff Squyres wrote:

[about -D not working with xlf]


If we care, it is easy enough to add a configure test to
figure this kind of stuff out.  


Might be worth logging a bug with the autotools/autoconf
people on this (if it's not already there), it's been
mentioned recently on their lists as something they
should look at doing better:

http://old.nabble.com/Re:-Can't-use-Fortran-90-95-compiler-for-F77-p26209677.html

cheers,
Chris
--
 Chris Samuel  :  http://www.csamuel.org/  :  Melbourne, VIC

Re: [OMPI devel] kernel 2.6.23 vs 2.6.24 - communication/wait times

[Chris Samuel]

On 10/04/10 15:12, Bogdan Costescu wrote:


Have there been any process scheduler changes in the newer kernels ?


Are there ever kernels where that doesn't get tweaked ? ;-)


I'm not sure that they could explain four orders of magnitude
differences though...


One idea that comes to mind would be to run the child processes
under strace -c as that will monitor all the system calls and
report how long is spent in which.   By running a comparison
with 2.6.23 and 2.6.24 then you might get a pointer to which
syscall(s) are taking longer.

Alternatively if you want to get fancy then you could play
with doing a git bisection between 2.6.23 and 2.6.24 to track
down the commit that introduces the regression.

To be honest it'd be interesting to see whether the issue still
manifests on a recent kernel though, if so then perhaps we might
be able to get the kernel developers interested (though they will
likely ask for a bisection too).

cheers!
Chris
--
 Chris Samuel  :  http://www.csamuel.org/  :  Melbourne, VIC

Re: [OMPI devel] kernel 2.6.23 vs 2.6.24 - communication/wait times

[Chris Samuel]

On 10/04/10 06:59, Oliver Geisler wrote:


This is the results of skampi pt2pt, first with shared memory allowed,
second shared memory excluded.


For what it's worth I can't replicate those results on an AMD Shanghai
cluster running a 2.6.32 kernel and Open-MPI 1.4.1.

Here is what I see (run under Torque, selecting 2 cores on the same
node, so no need to specify -np):

$ mpirun --mca btl self,sm,tcp  ./skampi -i ski/skampi_pt2pt.ski

# begin result "Pingpong_Send_Recv"
count= 14   2.0   0.0   16   2.0   1.8
count= 28   2.1   0.0   16   2.1   1.8
count= 3   12   2.1   0.18   2.0   2.0
count= 4   16   2.1   0.18   2.0   2.0
count= 6   24   2.0   0.0   16   2.0   1.8
count= 8   32   2.9   0.0   16   2.7   2.4
count= 11   44   2.3   0.1   16   2.2   2.0
count= 16   64   2.2   0.1   16   2.1   2.0
count= 23   92   2.7   0.2   16   2.6   2.1
count= 32  128   2.5   0.1   16   2.5   2.1
count= 45  180   3.0   0.0   16   2.8   2.6
count= 64  256   3.1   0.08   3.0   2.5
count= 91  364   3.1   0.08   3.0   3.0
count= 128  512   3.4   0.2   16   3.3   3.0
count= 181  724   4.1   0.0   16   4.0   4.1
count= 256 1024   5.0   0.08   4.5   4.5
count= 362 1448   6.0   0.0   16   5.8   5.7
count= 512 2048   7.7   0.1   16   7.3   7.6
count= 724 2896  10.0   0.08  10.0   9.8
count= 1024 4096  12.3   0.1   16  12.1  12.0
count= 1448 5792  13.8   0.28  13.5  13.4
count= 2048 8192  18.1   0.0   16  17.9  18.1
count= 289611584  25.0   0.0   16  24.9  25.0
count= 409616384  34.2   0.1   16  34.0  34.2
# end result "Pingpong_Send_Recv"
# duration = 0.00 sec

mpirun --mca btl tcp,self  ./skampi -i ski/skampi_pt2pt.ski

# begin result "Pingpong_Send_Recv"
count= 14  21.2   1.0   16  20.1  17.8
count= 28  20.8   1.0   16  20.6  16.7
count= 3   12  20.2   0.9   16  19.0  17.1
count= 4   16  19.9   1.0   16  19.0  17.0
count= 6   24  21.1   1.1   16  20.6  17.0
count= 8   32  20.0   1.0   16  18.8  17.1
count= 11   44  20.9   0.8   16  20.0  17.1
count= 16   64  21.7   1.1   16  20.5  17.6
count= 23   92  21.7   1.0   16  20.0  18.5
count= 32  128  21.6   1.0   16  20.5  18.5
count= 45  180  22.0   1.0   16  20.9  19.0
count= 64  256  21.8   0.7   16  20.5  20.2
count= 91  364  20.5   0.3   16  19.8  19.1
count= 128  512  18.5   0.38  17.5  18.1
count= 181  724  19.3   0.28  19.1  19.0
count= 256 1024  20.3   0.3   16  19.7  20.0
count= 362 1448  22.1   0.3   16  21.2  21.4
count= 512 2048  24.2   0.3   16  23.7  23.2
count= 724 2896  24.8   0.58  24.0  24.0
count= 1024 4096  26.8   0.2   16  26.1  26.3
count= 1448 5792  31.6   0.3   16  30.4  31.5
count= 2048 8192  38.0   0.6   16  37.3  37.1
count= 289611584  52.1   1.4   16  49.1  50.8
count= 409616384  93.8   1.1   16  81.1  91.5
# end result "Pingpong_Send_Recv"
# duration = 0.02 sec

cheers,
Chris
--
 Chris Samuel  :  http://www.csamuel.org/  :  Melbourne, VIC

[OMPI devel] OMPI 1.5 twitter notification plugin probably broken by switch to OAUTH

[Chris Samuel]

Hi folks,

Looking at the code for the Twitter notifier in OMPI 1.5
and seeing its use of HTTP basic authentication I would
suggest that it may be non-functional due to Twitters
switch to purely OAUTH based authentication for their API.

I'm trying to test it out here but I'm at a bit of a loss
to find a trivial command line to trigger a notifier, even
the syslog one!  I've Googled and grep'd with no joy..

For instance:

chris@quad:/tmp/ompi$ ./bin/mpirun --mca notifier syslog --mca 
notifier_threshold_severity 99 --mca 
notifier_base_verbose 99  /bin/doesnotexist
[quad:18429] mca: base: components_open: Looking for notifier components
[quad:18429] mca: base: components_open: opening notifier components
[quad:18429] mca: base: components_open: found loaded component syslog
[quad:18429] mca: base: components_open: component syslog has no register 
function
[quad:18429] mca: base: components_open: component syslog open function 
successful
[quad:18429] mca:base:select: Auto-selecting notifier components
[quad:18429] mca:base:select:(notifier) Querying component [syslog]
[quad:18429] mca:base:select:(notifier) Query of component [syslog] set 
priority to 1
[quad:18429] mca:base:select:(notifier) Selected component [syslog]
--
mpirun was unable to launch the specified application as it could not access
or execute an executable:

Executable: /bin/doesnotexist
Node: quad

while attempting to start process rank 0.
--
[quad:18429] mca: base: close: component syslog closed
[quad:18429] mca: base: close: unloading component syslog

But nothing appears in the syslog.

Ideas ?

cheers!
Chris
-- 
Christopher Samuel  Senior Systems Administrator
VLSCI - Victorian Life Sciences Computational Initiative
Email: sam...@unimelb.edu.au Phone: +61 (0)3 903 55545
  http://www.vlsci.unimelb.edu.au/

[OMPI devel] Enable issue tracker for ompi-www repo?

[Chris Samuel]

Hi folks,

I was looking to file an issue against the website for the FAQ about XRC 
support (given it was disabled in issue #4087) but it doesn't appear to be 
enabled.   Is that just an oversight or is there a different way preferred?

All the best,
Chris
-- 
 Christopher SamuelSenior Systems Administrator
 Melbourne Bioinformatics - The University of Melbourne
 Email: sam...@unimelb.edu.au Phone: +61 (0)3 903 55545

___
devel mailing list
devel@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/devel

Re: [OMPI devel] Enable issue tracker for ompi-www repo?

[Chris Samuel]

On Sunday, 5 November 2017 12:07:52 AM AEDT r...@open-mpi.org wrote:

> It was just an oversight - I have turned on the issue tracker, so feel free
> to post, or a PR is also welcome

Thanks Ralph!   Unfortunately I don't feel I understand enough about 
background of the issue to say much more about it other than "XRC is no longer 
available as of August 2017" otherwise I would try a PR.

-- 
 Christopher SamuelSenior Systems Administrator
 Melbourne Bioinformatics - The University of Melbourne
 Email: sam...@unimelb.edu.au Phone: +61 (0)3 903 55545

___
devel mailing list
devel@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/devel

Re: [OMPI devel] RFC: Remove embedded libltdl

[Chris Samuel]

On Mon, 2 Feb 2015 11:35:40 AM Jeff Squyres wrote:

> Ah -- the point being that this is not an issue related to the libltdl work.

Sorry - I saw the request to test the tarball and tried it out, missed the 
significance of the subject. :-/

-- 
 Christopher SamuelSenior Systems Administrator
 VLSCI - Victorian Life Sciences Computation Initiative
 Email: sam...@unimelb.edu.au Phone: +61 (0)3 903 55545
 http://www.vlsci.org.au/  http://twitter.com/vlsci

Re: [OMPI devel] [OMPI commits] Git: open-mpi/ompi branch master updated. dev-1731-g8e30579

[Chris Samuel]

On Sat, 16 May 2015 12:49:51 PM Jeff Squyres wrote:

> Linux / RHEL 6.5 / 2.6.32 kernel (this is clearly in seconds):
>
> $ sysctl net.ipv4.tcp_keepalive_time
> net.ipv4.tcp_keepalive_time = 1800

I suspect that's a local customisation, all Linux systems I've got access to 
(including RHEL 6.4/6.5/6.6) report:

net.ipv4.tcp_keepalive_time = 7200

All the best,
Chris
-- 
 Christopher SamuelSenior Systems Administrator
 VLSCI - Victorian Life Sciences Computation Initiative
 Email: sam...@unimelb.edu.au Phone: +61 (0)3 903 55545
 http://www.vlsci.org.au/  http://twitter.com/vlsci

Re: [OMPI devel] [OMPI commits] Git: open-mpi/ompi branch master updated. dev-1731-g8e30579

[Chris Samuel]

On Sat, 16 May 2015 02:59:35 PM Paul Hargrove wrote:

> I didn't find OpenBSD or Solaris docs ("grep -rl TCP_KEEP /usr/share/man"
> didn't find any matches).

This seems to document it for an unspecified version of Solaris:

http://docs.oracle.com/cd/E19120-01/open.solaris/819-2724/fsvdg/index.html

For OpenBSD this bugzilla entry for Firefox from early last year claims they 
are only available via sysctl options there, not via setsockopt():

https://bugzilla.mozilla.org/show_bug.cgi?id=970550#c8

There is some (meagre) documentation of those options here:

http://nixdoc.net/man-pages/OpenBSD/sysctl.3.html

The last (documented) change on the OpenBSD site was 3.5, saying:

http://www.openbsd.org/plus35.html

# Reset the TCP keepalive timer to tcp.keepidle (normally four hours)
# after the three-way handshake completes. (syncache sets it to
# tcp.keepinittime, normally 150 seconds).

Hope that helps!

All the best,
Chris
-- 
 Christopher SamuelSenior Systems Administrator
 VLSCI - Victorian Life Sciences Computation Initiative
 Email: sam...@unimelb.edu.au Phone: +61 (0)3 903 55545
 http://www.vlsci.org.au/  http://twitter.com/vlsci

Re: [OMPI devel] 1.8.7rc1 testing results

[Chris Samuel]

Hi Gilles,

On Mon, 13 Jul 2015 03:16:57 PM Gilles Gouaillardet wrote:

> i made ConnectX XRC (aka XRC) and ConnectIb XRC (aka XRC domains) exclusive,
> so yes, you got the desired behavior.

Is there a tarball I could test on our x86 systems please?  We are tied to the 
OFED in RHEL due to having BG/Q in the mix on our IB fabrics.

All the best,
Chris
-- 
 Christopher SamuelSenior Systems Administrator
 VLSCI - Victorian Life Sciences Computation Initiative
 Email: sam...@unimelb.edu.au Phone: +61 (0)3 903 55545
 http://www.vlsci.org.au/  http://twitter.com/vlsci

Re: [OMPI devel] 1.8.7rc1 testing results

[Chris Samuel]

On Mon, 13 Jul 2015 05:17:29 PM Gilles Gouaillardet wrote:

> Hi Chris,

Hi Gilles,

> i pushed my tarball into a gist :

Thanks for that, I can confirm on our two x86-64 RHEL 6.6 boxes (one circa
2010, one circa 2013) with their included OFED I see:

checking if ConnectX XRC support is enabled... no
checking if ConnectIB XRC support is enabled... yes

I also got this set of (unrelated) compiler warnings too:

  FCLD libmpi_usempif08.la
[...benign symlink related Slurm libpmi.la warnings elided...]
ifort: command line remark #10010: option '-pthread' is deprecated and will be 
removed in a future release. See '-help deprecated'
ld: Warning: alignment 4 of symbol `ompi_f08_mpi_logical8' in .libs/constants.o 
is smaller than 8 in 
./.libs/libforce_usempif08_internal_modules_to_be_built.a(mpi-f08-types.o)
ld: Warning: alignment 4 of symbol `ompi_f08_mpi_logical4' in .libs/constants.o 
is smaller than 8 in 
./.libs/libforce_usempif08_internal_modules_to_be_built.a(mpi-f08-types.o)
ld: Warning: alignment 4 of symbol `ompi_f08_mpi_logical2' in .libs/constants.o 
is smaller than 8 in 
./.libs/libforce_usempif08_internal_modules_to_be_built.a(mpi-f08-types.o)
ld: Warning: alignment 4 of symbol `ompi_f08_mpi_logical1' in .libs/constants.o 
is smaller than 8 in 
./.libs/libforce_usempif08_internal_modules_to_be_built.a(mpi-f08-types.o)
ld: Warning: alignment 4 of symbol `ompi_f08_mpi_real2' in .libs/constants.o is 
smaller than 8 in 
./.libs/libforce_usempif08_internal_modules_to_be_built.a(mpi-f08-types.o)
ld: Warning: alignment 4 of symbol `ompi_f08_mpi_2double_complex' in 
.libs/constants.o is smaller than 8 in 
./.libs/libforce_usempif08_internal_modules_to_be_built.a(mpi-f08-types.o)
ld: Warning: alignment 4 of symbol `ompi_f08_mpi_2complex' in .libs/constants.o 
is smaller than 8 in 
./.libs/libforce_usempif08_internal_modules_to_be_built.a(mpi-f08-types.o)
ld: Warning: alignment 4 of symbol `ompi_f08_mpi_2integer' in .libs/constants.o 
is smaller than 8 in 
./.libs/libforce_usempif08_internal_modules_to_be_built.a(mpi-f08-types.o)
ld: Warning: alignment 4 of symbol `ompi_f08_mpi_2double_precision' in 
.libs/constants.o is smaller than 8 in 
./.libs/libforce_usempif08_internal_modules_to_be_built.a(mpi-f08-types.o)
ld: Warning: alignment 4 of symbol `ompi_f08_mpi_2real' in .libs/constants.o is 
smaller than 8 in 
./.libs/libforce_usempif08_internal_modules_to_be_built.a(mpi-f08-types.o)
ld: Warning: alignment 4 of symbol `ompi_f08_mpi_double_complex' in 
.libs/constants.o is smaller than 8 in 
./.libs/libforce_usempif08_internal_modules_to_be_built.a(mpi-f08-types.o)
ld: Warning: alignment 4 of symbol `ompi_f08_mpi_complex32' in 
.libs/constants.o is smaller than 8 in 
./.libs/libforce_usempif08_internal_modules_to_be_built.a(mpi-f08-types.o)
ld: Warning: alignment 4 of symbol `ompi_f08_mpi_complex16' in 
.libs/constants.o is smaller than 8 in 
./.libs/libforce_usempif08_internal_modules_to_be_built.a(mpi-f08-types.o)
ld: Warning: alignment 4 of symbol `ompi_f08_mpi_complex8' in .libs/constants.o 
is smaller than 8 in 
./.libs/libforce_usempif08_internal_modules_to_be_built.a(mpi-f08-types.o)
ld: Warning: alignment 4 of symbol `ompi_f08_mpi_complex' in .libs/constants.o 
is smaller than 8 in 
./.libs/libforce_usempif08_internal_modules_to_be_built.a(mpi-f08-types.o)
ld: Warning: alignment 4 of symbol `ompi_f08_mpi_double_precision' in 
.libs/constants.o is smaller than 8 in 
./.libs/libforce_usempif08_internal_modules_to_be_built.a(mpi-f08-types.o)
ld: Warning: alignment 4 of symbol `ompi_f08_mpi_real16' in .libs/constants.o 
is smaller than 8 in 
./.libs/libforce_usempif08_internal_modules_to_be_built.a(mpi-f08-types.o)
ld: Warning: alignment 4 of symbol `ompi_f08_mpi_real8' in .libs/constants.o is 
smaller than 8 in 
./.libs/libforce_usempif08_internal_modules_to_be_built.a(mpi-f08-types.o)
ld: Warning: alignment 4 of symbol `ompi_f08_mpi_real4' in .libs/constants.o is 
smaller than 8 in 
./.libs/libforce_usempif08_internal_modules_to_be_built.a(mpi-f08-types.o)
ld: Warning: alignment 4 of symbol `ompi_f08_mpi_real' in .libs/constants.o is 
smaller than 8 in 
./.libs/libforce_usempif08_internal_modules_to_be_built.a(mpi-f08-types.o)
ld: Warning: alignment 4 of symbol `ompi_f08_mpi_integer16' in 
.libs/constants.o is smaller than 8 in 
./.libs/libforce_usempif08_internal_modules_to_be_built.a(mpi-f08-types.o)
ld: Warning: alignment 4 of symbol `ompi_f08_mpi_integer8' in .libs/constants.o 
is smaller than 8 in 
./.libs/libforce_usempif08_internal_modules_to_be_built.a(mpi-f08-types.o)
ld: Warning: alignment 4 of symbol `ompi_f08_mpi_integer4' in .libs/constants.o 
is smaller than 8 in 
./.libs/libforce_usempif08_internal_modules_to_be_built.a(mpi-f08-types.o)
ld: Warning: alignment 4 of symbol `ompi_f08_mpi_integer2' in .libs/constants.o 
is smaller than 8 in 
./.libs/libforce_usempif08_internal_modules_to_be_built.a(mpi-f08-types.o)
ld: Warning: alignment 4 of symbol `ompi_f08_mpi_integer1' in .libs/const