Re: [OMPI users] Fortran and MPI-3 shared memory

[Gilles Gouaillardet]

Tom,


regardless the (lack of) memory model in Fortran, there is an error in 
testmpi3.f90


shar_mem is declared as an integer, and hence is not in the shared memory.

i attached my version of testmpi3.f90, which behaves just like the C 
version,


at least when compiled with -g -O0 and with Open MPI master

/* i replaced shar_mem with fptr_mem */


Cheers,


Gilles




On 10/26/2016 3:29 AM, Tom Rosmond wrote:

All:

I am trying to understand the use of the shared memory features of 
MPI-3 that allow direct sharing of the memory space of on-node 
processes.  Attached are 2 small test programs, one written in C 
(testmpi3.c), the other F95 (testmpi3.f90) .  They are solving the 
identical 'halo' exchange problem.  'testmpi3.c' is a simplified 
version of an example program from a presentation by Mark Lubin of 
Intel.  I wrote 'testmpi3.f90' to mimic the C version.


 Also attached are 2 text files of the compile, execution, and output 
of the respective programs:


CC_testmpi3.txt
F95_testmpi3.txt

Note: All 4 files are contained in the attached 'testmpi3.tar.gz'.

Comparing the outputs of each version, it is clear that the shared 
memory copies in 'testmpi3.c' are working correctly, but not in 
'testmpi3.f90'.  As far as I can tell, the 2 programs are equivalent 
up to line 134 of 'testmpi3.c' and lines 97-101 of 'testmpi3.f90'. I 
thought the calls to 'c_f_pointer' would produce Fortran pointers that 
would access the correct shared memory addresses as the C-pointers do 
in 'testmpi3.c', but clearly that isn't happening. Can anyone explain 
why not, and what is needed to make this happen. Any suggestions are 
welcome.


My environment:
 Scientific Linux 6.8
 INTEL FORTRAN and ICC version 15.0.2.164
 OPEN-MPI 2.0.1


T. Rosmond


___
users mailing list
users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/users


  program test
!
! This program does 'halo' exchange

  use mpi_f08
  USE, INTRINSIC ::  ISO_C_BINDING, ONLY : C_PTR, C_F_POINTER

  implicit none

  integer ierr,nproc,nprocsh,lenname
  integer partners(0:1)
  integer maps_shm(0:1)
  integer irank,ir
  integer icolor,iranksh,idisp,j,jr,ii,numnodes,nx

  integer off_node_partners
  integer on_node_partners
  integer halo_left(0:0)
  integer halo_right(0:0)

  integer(kind=MPI_ADDRESS_KIND) :: isizew
  type(mpi_status):: istat
  type(c_ptr) ::  shar_cptr
  type(mpi_comm) :: mpicomm,shmcomm
  type(mpi_win) :: iwin
  type(mpi_info) :: info
  type(mpi_group) :: world_group,share_group

  integer, pointer :: fptr_left(:)
  integer, pointer :: fptr_right(:)
  integer, pointer :: fptr_mem(:)
  integer, allocatable :: shape(:)

  mpicomm= mpi_comm_world
  info= mpi_info_null

  call MPI_Init(ierr)
  call MPI_Comm_rank(mpicomm,irank,ierr)
  call MPI_Comm_size(mpicomm,nproc,ierr)
  ir= irank+1
!
! Define halo partners ring with periodic BC
!
  partners(0)= irank-1
  partners(1)= irank+1
  if(irank.eq.0) partners(0)= nproc-1
  if(irank.eq.nproc-1) partners(1)= 0
!
! Am runing on fully NUMA system, so split mpicomm explictly to mimic
! cluster nodes.  Here I split into 4 equally sized nodes 

  numnodes= 4

   icolor= (irank*numnodes+1)/nproc
   call mpi_comm_split(mpicomm,icolor,0,shmcomm,ierr)
!
   call mpi_comm_group(mpicomm,world_group,ierr)
   call mpi_comm_group(shmcomm,share_group,ierr)
!
   call 
mpi_group_translate_ranks(world_group,2,partners,share_group,maps_shm,ierr)
!
  off_node_partners=0
  on_node_partners=0
  do j=0,1
   if(maps_shm(j).eq.MPI_UNDEFINED) then
 off_node_partners= off_node_partners+1
   else
 on_node_partners= on_node_partners+1
   endif
  enddo

  if(on_node_partners.eq.0) then 
   isizew=4
   idisp=1
  else
   isizew=4 
   idisp=1
  endif

  allocate(shape(1))
  shape=(/1/)

! allocate shared memory windows

  if(on_node_partners.ne.0) then

  call 
mpi_win_allocate_shared(isizew,idisp,info,shmcomm,shar_cptr,iwin,ierr)
  call c_f_pointer(shar_cptr,fptr_mem,shape)

! if(on_node_partners.ne.0) then

  do j=0,1
  
   if(maps_shm(j).ne.MPI_UNDEFINED) then
call mpi_win_shared_query(iwin,maps_shm(j),isizew,idisp,shar_cptr,ierr)
   endif

 if(j.eq.0) then
   call c_f_pointer(shar_cptr,fptr_left,shape)
 else
   call c_f_pointer(shar_cptr,fptr_right,shape)
 endif

  enddo
  endif
!
  call mpi_win_fence(0,iwin,ierr)
!
! fill shared memory with data

   fpt_mem(1) = (irank+1)*100

  call mpi_win_fence(0,iwin,ierr)

  do j=0,1
   jr= 1-j
   if(maps_shm(j) .ne. MPI_UNDEFINED) then ! partner j is on the same node 
if(j.eq.0) then
  halo_left= fptr_left  !  load from MPI-3/SHM ops
else
  

Re: [OMPI users] Unable to compile OpenMPI 1.10.3 with CUDA

[Sylvain Jeaugey]

I guess --with-cuda is disabling the default CUDA path which is 
/usr/local/cuda. So you should either not set --with-cuda or set 
--with-cuda $CUDA_HOME (no include).


Sylvain

On 10/27/2016 03:23 PM, Craig tierney wrote:

Hello,

I am trying to build OpenMPI 1.10.3 with CUDA but I am unable to build 
the library that will allow me to use IPC on a node or GDR between 
nodes.   I have tried with with 1.10.4 and 2.0.1 and have the same 
problems.  Here is my build script:


---
#!/bin/bash

export OPENMPI_VERSION=1.10.3
export BASEDIR=/tmp/mpi_testing/
export CUDA_HOME=/usr/local/cuda
export PATH=$CUDA_HOME/bin/:$PATH
export LD_LIBRARY_PATH=$CUDA_HOME/lib64:$LD_LIBRARY_PATH
export MPI_HOME=$BASEDIR/openmpi-$OPENMPI_VERSION

which nvcc
nvcc --version

tar -zxf openmpi-$OPENMPI_VERSION.tar.gz
cd openmpi-$OPENMPI_VERSION

./configure --prefix=$MPI_HOME --with-cuda=$CUDA_HOME/include > 
config.out 2>&1


make -j > build.out 2>&1
make install >> build.out 2>&1
---

From the docs, it appears that I should not have to set anything but 
--with-cuda since my CUDA is in /usr/local/cuda. However, I appended 
/usr/local/cuda/include just in case when the first way didn't work.


From the output in config.log, I see that cuda.h is not found.  When 
the tests are called there is no extra include flag added to specify 
the /usr/local/cuda/include path.


With the resulting build, I test for CUDA and GDR with ompi_info.  
Results are:


testuser@dgx-1:~/temp$ /tmp/mpi_testing/openmpi-1.10.3/bin/ompi_info 
 | grep cuda
 MCA btl: smcuda (MCA v2.0.0, API v2.0.0, Component 
v1.10.3)

MCA coll: cuda (MCA v2.0.0, API v2.0.0, Component v1.10.3)
testuser@dgx-1:~/temp$ /tmp/mpi_testing/openmpi-1.10.3/bin/ompi_info 
 | grep gdr

testuser@dgx-1:~/temp$

Configure and build logs are attached.

Thanks,
Craig



___
users mailing list
users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/users



---
This email message is for the sole use of the intended recipient(s) and may 
contain
confidential information.  Any unauthorized review, use, disclosure or 
distribution
is prohibited.  If you are not the intended recipient, please contact the 
sender by
reply email and destroy all copies of the original message.
---
___
users mailing list
users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

Re: [OMPI users] Slurm binding not propagated to MPI jobs

[Andy Riebs]

  
  
Hi Ralph,
I haven't played around in this code, so I'll flip the question
  over to the Slurm list, and report back here when I learn
  anything.
Cheers
Andy

On 10/27/2016 01:44 PM,
  r...@open-mpi.org wrote:


  
  Sigh - of course it wouldn’t be simple :-(
  
  
  All right, let’s suppose we look for SLURM_CPU_BIND:
  
  
  * if it includes the word “none”, then we know the
user specified that they don’t want us to bind
  
  
  * if it includes the word mask_cpu, then we have to
check the value of that option.
  
  
  
* If it is all F’s, then they didn’t specify a
  binding and we should do our thing.


* If it is anything else, then we assume they
  _did_ specify a binding, and we leave it alone
  
  
  
  Would that make sense? Is there anything else that
could be in that envar which would trip us up?
  
  
  

  
On Oct 27, 2016, at 10:37 AM, Andy Riebs  wrote:


  
Yes, they still exist:
$ srun --ntasks-per-node=2 -N1 env | grep
  BIND | sort -u
  SLURM_CPU_BIND_LIST=0x
  SLURM_CPU_BIND=quiet,mask_cpu:0x
  SLURM_CPU_BIND_TYPE=mask_cpu:
  SLURM_CPU_BIND_VERBOSE=quiet

Here are the relevant Slurm configuration
  options that could conceivably change the behavior
  from system to system:

SelectType  = select/cons_res
  SelectTypeParameters    = CR_CPU
  


On 10/27/2016 01:17 PM, r...@open-mpi.org
  wrote:

 And if there is no --cpu_bind on
  the cmd line? Do these not exist?
  

  
On Oct 27, 2016, at 10:14 AM, Andy
  Riebs 
  wrote:

Hi
Ralph,
  
  I
think I've found the magic keys...
  
  $ srun
--ntasks-per-node=2 -N1 --cpu_bind=none env
| grep BIND
  SLURM_CPU_BIND_VERBOSE=quiet
  SLURM_CPU_BIND_TYPE=none
  SLURM_CPU_BIND_LIST=
  SLURM_CPU_BIND=quiet,none
  SLURM_CPU_BIND_VERBOSE=quiet
  SLURM_CPU_BIND_TYPE=none
  SLURM_CPU_BIND_LIST=
  SLURM_CPU_BIND=quiet,none
  $ srun
--ntasks-per-node=2 -N1 --cpu_bind=core env
| grep BIND
  SLURM_CPU_BIND_VERBOSE=quiet
  SLURM_CPU_BIND_TYPE=mask_cpu:
  SLURM_CPU_BIND_LIST=0x,0x
  SLURM_CPU_BIND=quiet,mask_cpu:0x,0x
  SLURM_CPU_BIND_VERBOSE=quiet
  SLURM_CPU_BIND_TYPE=mask_cpu:
  SLURM_CPU_BIND_LIST=0x,0x
  SLURM_CPU_BIND=quiet,mask_cpu:0x,0x
  
  Andy
  
  On
10/27/2016 11:57 AM, r...@open-mpi.org wrote:
  Hey Andy

Is there a SLURM envar that would tell us
the binding option from the srun cmd line?
We automatically bind when direct launched
due to user complaints of poor performance
if we don’t. If the user specifies a
binding option, then we detect that we were
already bound and don’t do it.

However, if the user specifies that they not
be bound, then we think they simply
didn’t specify anything - and that
isn’t the case. If we can see
something that tells us “they
explicitly said not to do it”, then we
can avoid the situation.

Ralph
   

Re: [OMPI users] Slurm binding not propagated to MPI jobs

[r...@open-mpi.org]

Sigh - of course it wouldn’t be simple :-(

All right, let’s suppose we look for SLURM_CPU_BIND:

* if it includes the word “none”, then we know the user specified that they 
don’t want us to bind

* if it includes the word mask_cpu, then we have to check the value of that 
option.

* If it is all F’s, then they didn’t specify a binding and we should do our 
thing.

* If it is anything else, then we assume they _did_ specify a binding, and we 
leave it alone

Would that make sense? Is there anything else that could be in that envar which 
would trip us up?


> On Oct 27, 2016, at 10:37 AM, Andy Riebs  wrote:
> 
> Yes, they still exist:
> 
> $ srun --ntasks-per-node=2 -N1 env | grep BIND | sort -u
> SLURM_CPU_BIND_LIST=0x
> SLURM_CPU_BIND=quiet,mask_cpu:0x
> SLURM_CPU_BIND_TYPE=mask_cpu:
> SLURM_CPU_BIND_VERBOSE=quiet
> Here are the relevant Slurm configuration options that could conceivably 
> change the behavior from system to system:
> SelectType  = select/cons_res
> SelectTypeParameters= CR_CPU
> 
> 
> On 10/27/2016 01:17 PM, r...@open-mpi.org  wrote:
>> And if there is no --cpu_bind on the cmd line? Do these not exist?
>> 
>>> On Oct 27, 2016, at 10:14 AM, Andy Riebs >> > wrote:
>>> 
>>> Hi Ralph,
>>> 
>>> I think I've found the magic keys...
>>> 
>>> $ srun --ntasks-per-node=2 -N1 --cpu_bind=none env | grep BIND
>>> SLURM_CPU_BIND_VERBOSE=quiet
>>> SLURM_CPU_BIND_TYPE=none
>>> SLURM_CPU_BIND_LIST=
>>> SLURM_CPU_BIND=quiet,none
>>> SLURM_CPU_BIND_VERBOSE=quiet
>>> SLURM_CPU_BIND_TYPE=none
>>> SLURM_CPU_BIND_LIST=
>>> SLURM_CPU_BIND=quiet,none
>>> $ srun --ntasks-per-node=2 -N1 --cpu_bind=core env | grep BIND
>>> SLURM_CPU_BIND_VERBOSE=quiet
>>> SLURM_CPU_BIND_TYPE=mask_cpu:
>>> SLURM_CPU_BIND_LIST=0x,0x
>>> SLURM_CPU_BIND=quiet,mask_cpu:0x,0x
>>> SLURM_CPU_BIND_VERBOSE=quiet
>>> SLURM_CPU_BIND_TYPE=mask_cpu:
>>> SLURM_CPU_BIND_LIST=0x,0x
>>> SLURM_CPU_BIND=quiet,mask_cpu:0x,0x
>>> 
>>> Andy
>>> 
>>> On 10/27/2016 11:57 AM, r...@open-mpi.org  wrote:
 Hey Andy
 
 Is there a SLURM envar that would tell us the binding option from the srun 
 cmd line? We automatically bind when direct launched due to user 
 complaints of poor performance if we don’t. If the user specifies a 
 binding option, then we detect that we were already bound and don’t 
 do it.
 
 However, if the user specifies that they not be bound, then we think they 
 simply didn’t specify anything - and that isn’t the case. If 
 we can see something that tells us “they explicitly said not to do 
 it”, then we can avoid the situation.
 
 Ralph
 
> On Oct 27, 2016, at 8:48 AM, Andy Riebs  > wrote:
> 
> Hi All,
> 
> We are running Open MPI version 1.10.2, built with support for Slurm 
> version 16.05.0. When a user specifies "--cpu_bind=none", MPI tries to 
> bind by core, which segv's if there are more processes than cores.
> 
> The user reports:
> 
> What I found is that
> 
> % srun --ntasks-per-node=8 --cpu_bind=none  \
> env SHMEM_SYMMETRIC_HEAP_SIZE=1024M bin/all2all.shmem.exe 0
> 
> will have the problem, but:
> 
> % srun --ntasks-per-node=8 --cpu_bind=none  \
> env SHMEM_SYMMETRIC_HEAP_SIZE=1024M ./bindit.sh bin/all2all.shmem.exe > 0
> 
> Will run as expected and print out the usage message because I 
> didn’t provide the right arguments to the code.
> 
> So, it appears that the binding has something to do with the issue. My 
> binding script is as follows:
> 
> % cat bindit.sh
> #!/bin/bash
> 
> #echo SLURM_LOCALID=$SLURM_LOCALID
> 
> stride=1
> 
> if [ ! -z "$SLURM_LOCALID" ]; then
>   let bindCPU=$SLURM_LOCALID*$stride
>   exec numactl --membind=0 --physcpubind=$bindCPU $*
> fi
> 
> $*
> 
> %
> 
> 
> -- 
> Andy Riebs
> andy.ri...@hpe.com 
> Hewlett-Packard Enterprise
> High Performance Computing Software Engineering
> +1 404 648 9024
> My opinions are not necessarily those of HPE
>May the source be with you!
> 
> ___
> users mailing list
> users@lists.open-mpi.org 
> https://rfd.newmexicoconsortium.org/mailman/listinfo/users 
> 
 ___
 users mailing list
 users@lists.open-mpi.org 
 https://rfd.newmexicoconsortium.org/mailman/listinfo/users 
 
>>> 
>>> 

Re: [OMPI users] Slurm binding not propagated to MPI jobs

[Andy Riebs]

  
  
Yes, they still exist:
$ srun --ntasks-per-node=2 -N1 env | grep BIND | sort -u
  SLURM_CPU_BIND_LIST=0x
  SLURM_CPU_BIND=quiet,mask_cpu:0x
  SLURM_CPU_BIND_TYPE=mask_cpu:
  SLURM_CPU_BIND_VERBOSE=quiet

Here are the relevant Slurm configuration options that could
  conceivably change the behavior from system to system:

SelectType  = select/cons_res
  SelectTypeParameters    = CR_CPU
  


On 10/27/2016 01:17 PM,
  r...@open-mpi.org wrote:


  
  And if there is no --cpu_bind on the cmd line? Do these not exist?
  

  
On Oct 27, 2016, at 10:14 AM, Andy Riebs  wrote:

Hi
Ralph,
  
  I
think I've found the magic keys...
  
  $
srun --ntasks-per-node=2 -N1 --cpu_bind=none env | grep
BIND
  SLURM_CPU_BIND_VERBOSE=quiet
  SLURM_CPU_BIND_TYPE=none
  SLURM_CPU_BIND_LIST=
  SLURM_CPU_BIND=quiet,none
  SLURM_CPU_BIND_VERBOSE=quiet
  SLURM_CPU_BIND_TYPE=none
  SLURM_CPU_BIND_LIST=
  SLURM_CPU_BIND=quiet,none
  $
srun --ntasks-per-node=2 -N1 --cpu_bind=core env | grep
BIND
  SLURM_CPU_BIND_VERBOSE=quiet
  SLURM_CPU_BIND_TYPE=mask_cpu:
  SLURM_CPU_BIND_LIST=0x,0x
  SLURM_CPU_BIND=quiet,mask_cpu:0x,0x
  SLURM_CPU_BIND_VERBOSE=quiet
  SLURM_CPU_BIND_TYPE=mask_cpu:
  SLURM_CPU_BIND_LIST=0x,0x
  SLURM_CPU_BIND=quiet,mask_cpu:0x,0x
  
  Andy
  
  On
10/27/2016 11:57 AM, r...@open-mpi.org wrote:
  Hey Andy

Is there a SLURM envar that would tell us the binding
option from the srun cmd line? We automatically bind
when direct launched due to user complaints of poor
performance if we don’t. If the user specifies a
binding option, then we detect that we were already
bound and don’t do it.

However, if the user specifies that they not be bound,
then we think they simply didn’t specify anything
- and that isn’t the case. If we can see
something that tells us “they explicitly said not
to do it”, then we can avoid the situation.

Ralph

On Oct 27, 2016, at
  8:48 AM, Andy Riebs 
  wrote:
  
  Hi All,
  
  We are running Open MPI version 1.10.2, built with
  support for Slurm version 16.05.0. When a user
  specifies "--cpu_bind=none", MPI tries to bind by
  core, which segv's if there are more processes than
  cores.
  
  The user reports:
  
  What I found is that
  
  % srun --ntasks-per-node=8 --cpu_bind=none  \
  env SHMEM_SYMMETRIC_HEAP_SIZE=1024M
  bin/all2all.shmem.exe 0
  
  will have the problem, but:
  
  % srun --ntasks-per-node=8 --cpu_bind=none  \
  env SHMEM_SYMMETRIC_HEAP_SIZE=1024M ./bindit.sh
  bin/all2all.shmem.exe 0
  
  Will run as expected and print out the usage message
  because I didn’t provide the right arguments to
  the code.
  
  So, it appears that the binding has something to do
  with the issue. My binding script is as follows:
  
  % cat bindit.sh
  #!/bin/bash
  
  #echo SLURM_LOCALID=$SLURM_LOCALID
  
  stride=1
  
  if [ ! -z "$SLURM_LOCALID" ]; then
    let bindCPU=$SLURM_LOCALID*$stride
    exec numactl --membind=0 --physcpubind=$bindCPU $*
  fi
  
  $*
  
  %
  
  
  -- 
  Andy Riebs
  andy.ri...@hpe.com
  Hewlett-Packard Enterprise
  High Performance Computing Software Engineering
  +1 404 648 9024
  My opinions are not necessarily those of HPE

[OMPI users] MPI_Reduce related questions

[MM]

Hello,

Given mpi nodes 0  N-1, 0 being root, master node.
and trying to determine the maximum value of a function over a large
range of values of its parameters,

What are the differences between if any:

1. At node i:
evaluate f for each of the values assigned to i of the parameters space.
calculate maximum of f
send this max back to root node, and calculate max of maxes.

2. Assuming all nodes handle the same number of values of the parameters of f:
evaluate f for each
reduce the array of f results back to root node using the MAX OP for e.g.
at root node, calculate the max of maxes

feels like doing rows then columns vs columns then rows in a matrix.

Basically, what can help me choose?

Performance is always factor 1. If same performance, then code
simplicity comes second

thanks,
___
users mailing list
users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

Re: [OMPI users] Slurm binding not propagated to MPI jobs

[r...@open-mpi.org]

And if there is no --cpu_bind on the cmd line? Do these not exist?

> On Oct 27, 2016, at 10:14 AM, Andy Riebs  wrote:
> 
> Hi Ralph,
> 
> I think I've found the magic keys...
> 
> $ srun --ntasks-per-node=2 -N1 --cpu_bind=none env | grep BIND
> SLURM_CPU_BIND_VERBOSE=quiet
> SLURM_CPU_BIND_TYPE=none
> SLURM_CPU_BIND_LIST=
> SLURM_CPU_BIND=quiet,none
> SLURM_CPU_BIND_VERBOSE=quiet
> SLURM_CPU_BIND_TYPE=none
> SLURM_CPU_BIND_LIST=
> SLURM_CPU_BIND=quiet,none
> $ srun --ntasks-per-node=2 -N1 --cpu_bind=core env | grep BIND
> SLURM_CPU_BIND_VERBOSE=quiet
> SLURM_CPU_BIND_TYPE=mask_cpu:
> SLURM_CPU_BIND_LIST=0x,0x
> SLURM_CPU_BIND=quiet,mask_cpu:0x,0x
> SLURM_CPU_BIND_VERBOSE=quiet
> SLURM_CPU_BIND_TYPE=mask_cpu:
> SLURM_CPU_BIND_LIST=0x,0x
> SLURM_CPU_BIND=quiet,mask_cpu:0x,0x
> 
> Andy
> 
> On 10/27/2016 11:57 AM, r...@open-mpi.org  wrote:
>> Hey Andy
>> 
>> Is there a SLURM envar that would tell us the binding option from the srun 
>> cmd line? We automatically bind when direct launched due to user complaints 
>> of poor performance if we don’t. If the user specifies a binding option, 
>> then we detect that we were already bound and don’t do it.
>> 
>> However, if the user specifies that they not be bound, then we think they 
>> simply didn’t specify anything - and that isn’t the case. If we can see 
>> something that tells us “they explicitly said not to do it”, then we can 
>> avoid the situation.
>> 
>> Ralph
>> 
>>> On Oct 27, 2016, at 8:48 AM, Andy Riebs >> > wrote:
>>> 
>>> Hi All,
>>> 
>>> We are running Open MPI version 1.10.2, built with support for Slurm 
>>> version 16.05.0. When a user specifies "--cpu_bind=none", MPI tries to bind 
>>> by core, which segv's if there are more processes than cores.
>>> 
>>> The user reports:
>>> 
>>> What I found is that
>>> 
>>> % srun --ntasks-per-node=8 --cpu_bind=none  \
>>> env SHMEM_SYMMETRIC_HEAP_SIZE=1024M bin/all2all.shmem.exe 0
>>> 
>>> will have the problem, but:
>>> 
>>> % srun --ntasks-per-node=8 --cpu_bind=none  \
>>> env SHMEM_SYMMETRIC_HEAP_SIZE=1024M ./bindit.sh bin/all2all.shmem.exe 0
>>> 
>>> Will run as expected and print out the usage message because I didn’t 
>>> provide the right arguments to the code.
>>> 
>>> So, it appears that the binding has something to do with the issue. My 
>>> binding script is as follows:
>>> 
>>> % cat bindit.sh
>>> #!/bin/bash
>>> 
>>> #echo SLURM_LOCALID=$SLURM_LOCALID
>>> 
>>> stride=1
>>> 
>>> if [ ! -z "$SLURM_LOCALID" ]; then
>>>   let bindCPU=$SLURM_LOCALID*$stride
>>>   exec numactl --membind=0 --physcpubind=$bindCPU $*
>>> fi
>>> 
>>> $*
>>> 
>>> %
>>> 
>>> 
>>> -- 
>>> Andy Riebs
>>> andy.ri...@hpe.com
>>> Hewlett-Packard Enterprise
>>> High Performance Computing Software Engineering
>>> +1 404 648 9024
>>> My opinions are not necessarily those of HPE
>>>May the source be with you!
>>> 
>>> ___
>>> users mailing list
>>> users@lists.open-mpi.org
>>> https://rfd.newmexicoconsortium.org/mailman/listinfo/users
>> ___
>> users mailing list
>> users@lists.open-mpi.org
>> https://rfd.newmexicoconsortium.org/mailman/listinfo/users
> 
> ___
> users mailing list
> users@lists.open-mpi.org 
> https://rfd.newmexicoconsortium.org/mailman/listinfo/users 
> 
___
users mailing list
users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

Re: [OMPI users] Slurm binding not propagated to MPI jobs

[Andy Riebs]

Hi Ralph,

I think I've found the magic keys...

$ srun --ntasks-per-node=2 -N1 --cpu_bind=none env | grep BIND
SLURM_CPU_BIND_VERBOSE=quiet
SLURM_CPU_BIND_TYPE=none
SLURM_CPU_BIND_LIST=
SLURM_CPU_BIND=quiet,none
SLURM_CPU_BIND_VERBOSE=quiet
SLURM_CPU_BIND_TYPE=none
SLURM_CPU_BIND_LIST=
SLURM_CPU_BIND=quiet,none
$ srun --ntasks-per-node=2 -N1 --cpu_bind=core env | grep BIND
SLURM_CPU_BIND_VERBOSE=quiet
SLURM_CPU_BIND_TYPE=mask_cpu:
SLURM_CPU_BIND_LIST=0x,0x
SLURM_CPU_BIND=quiet,mask_cpu:0x,0x
SLURM_CPU_BIND_VERBOSE=quiet
SLURM_CPU_BIND_TYPE=mask_cpu:
SLURM_CPU_BIND_LIST=0x,0x
SLURM_CPU_BIND=quiet,mask_cpu:0x,0x

Andy

On 10/27/2016 11:57 AM, r...@open-mpi.org wrote:

Hey Andy

Is there a SLURM envar that would tell us the binding option from the srun cmd 
line? We automatically bind when direct launched due to user complaints of poor 
performance if we don’t. If the user specifies a binding option, then we 
detect that we were already bound and don’t do it.

However, if the user specifies that they not be bound, then we think they 
simply didn’t specify anything - and that isn’t the case. If we can see 
something that tells us “they explicitly said not to do it”, then we can 
avoid the situation.

Ralph


On Oct 27, 2016, at 8:48 AM, Andy Riebs  wrote:

Hi All,

We are running Open MPI version 1.10.2, built with support for Slurm version 16.05.0. 
When a user specifies "--cpu_bind=none", MPI tries to bind by core, which 
segv's if there are more processes than cores.

The user reports:

What I found is that

% srun --ntasks-per-node=8 --cpu_bind=none  \
 env SHMEM_SYMMETRIC_HEAP_SIZE=1024M bin/all2all.shmem.exe 0

will have the problem, but:

% srun --ntasks-per-node=8 --cpu_bind=none  \
 env SHMEM_SYMMETRIC_HEAP_SIZE=1024M ./bindit.sh bin/all2all.shmem.exe 0

Will run as expected and print out the usage message because I didn’t provide 
the right arguments to the code.

So, it appears that the binding has something to do with the issue. My binding 
script is as follows:

% cat bindit.sh
#!/bin/bash

#echo SLURM_LOCALID=$SLURM_LOCALID

stride=1

if [ ! -z "$SLURM_LOCALID" ]; then
   let bindCPU=$SLURM_LOCALID*$stride
   exec numactl --membind=0 --physcpubind=$bindCPU $*
fi

$*

%


--
Andy Riebs
andy.ri...@hpe.com
Hewlett-Packard Enterprise
High Performance Computing Software Engineering
+1 404 648 9024
My opinions are not necessarily those of HPE
May the source be with you!

___
users mailing list
users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

___
users mailing list
users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/users


___
users mailing list
users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

Re: [OMPI users] Slurm binding not propagated to MPI jobs

[r...@open-mpi.org]

Hey Andy

Is there a SLURM envar that would tell us the binding option from the srun cmd 
line? We automatically bind when direct launched due to user complaints of poor 
performance if we don’t. If the user specifies a binding option, then we detect 
that we were already bound and don’t do it.

However, if the user specifies that they not be bound, then we think they 
simply didn’t specify anything - and that isn’t the case. If we can see 
something that tells us “they explicitly said not to do it”, then we can avoid 
the situation.

Ralph

> On Oct 27, 2016, at 8:48 AM, Andy Riebs  wrote:
> 
> Hi All,
> 
> We are running Open MPI version 1.10.2, built with support for Slurm version 
> 16.05.0. When a user specifies "--cpu_bind=none", MPI tries to bind by core, 
> which segv's if there are more processes than cores.
> 
> The user reports:
> 
> What I found is that
> 
> % srun --ntasks-per-node=8 --cpu_bind=none  \
> env SHMEM_SYMMETRIC_HEAP_SIZE=1024M bin/all2all.shmem.exe 0
> 
> will have the problem, but:
> 
> % srun --ntasks-per-node=8 --cpu_bind=none  \
> env SHMEM_SYMMETRIC_HEAP_SIZE=1024M ./bindit.sh bin/all2all.shmem.exe 0
> 
> Will run as expected and print out the usage message because I didn’t provide 
> the right arguments to the code.
> 
> So, it appears that the binding has something to do with the issue. My 
> binding script is as follows:
> 
> % cat bindit.sh
> #!/bin/bash
> 
> #echo SLURM_LOCALID=$SLURM_LOCALID
> 
> stride=1
> 
> if [ ! -z "$SLURM_LOCALID" ]; then
>   let bindCPU=$SLURM_LOCALID*$stride
>   exec numactl --membind=0 --physcpubind=$bindCPU $*
> fi
> 
> $*
> 
> %
> 
> 
> -- 
> Andy Riebs
> andy.ri...@hpe.com
> Hewlett-Packard Enterprise
> High Performance Computing Software Engineering
> +1 404 648 9024
> My opinions are not necessarily those of HPE
>May the source be with you!
> 
> ___
> users mailing list
> users@lists.open-mpi.org
> https://rfd.newmexicoconsortium.org/mailman/listinfo/users

___
users mailing list
users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

[OMPI users] Slurm binding not propagated to MPI jobs

[Andy Riebs]

Hi All,

We are running Open MPI version 1.10.2, built with support for Slurm 
version 16.05.0. When a user specifies "--cpu_bind=none", MPI tries to 
bind by core, which segv's if there are more processes than cores.


The user reports:

What I found is that

% srun --ntasks-per-node=8 --cpu_bind=none  \
 env SHMEM_SYMMETRIC_HEAP_SIZE=1024M bin/all2all.shmem.exe 0

will have the problem, but:

% srun --ntasks-per-node=8 --cpu_bind=none  \
 env SHMEM_SYMMETRIC_HEAP_SIZE=1024M ./bindit.sh bin/all2all.shmem.exe 0

Will run as expected and print out the usage message because I didn’t 
provide the right arguments to the code.


So, it appears that the binding has something to do with the issue. My 
binding script is as follows:


% cat bindit.sh
#!/bin/bash

#echo SLURM_LOCALID=$SLURM_LOCALID

stride=1

if [ ! -z "$SLURM_LOCALID" ]; then
   let bindCPU=$SLURM_LOCALID*$stride
   exec numactl --membind=0 --physcpubind=$bindCPU $*
fi

$*

%


--
Andy Riebs
andy.ri...@hpe.com
Hewlett-Packard Enterprise
High Performance Computing Software Engineering
+1 404 648 9024
My opinions are not necessarily those of HPE
May the source be with you!

___
users mailing list
users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

Re: [OMPI users] redeclared identifier for openmpi-v2.0.1-130-gb3a367d witj Sun C on Linux

[Jeff Squyres (jsquyres)]

This fix for this was just merged (we had previously fixed it in the v2.x 
branch, but neglected to also put it on the v2.0.x branch) -- it should be in 
tonight's tarball:

https://github.com/open-mpi/ompi/pull/2295



> On Oct 27, 2016, at 6:45 AM, Siegmar Gross 
>  wrote:
> 
> Hi,
> 
> I tried to install openmpi-v2.0.1-130-gb3a367d on my "SUSE Linux
> Enterprise Server 12.1 (x86_64)" with Sun C 5.14 beta. Unfortunately,
> I get the following error. I was able to build it with gcc-6.2.0.
> 
> 
> loki openmpi-v2.0.1-130-gb3a367d-Linux.x86_64.64_cc 124 tail -18 
> log.make.Linux.x86_64.64_cc
>  CC   cxx_glue.lo
> "../../../../openmpi-v2.0.1-130-gb3a367d/opal/include/opal/sys/amd64/atomic.h",
>  line 159: warning: parameter in inline asm statement unused: %3
> "../../../../openmpi-v2.0.1-130-gb3a367d/opal/include/opal/sys/amd64/atomic.h",
>  line 205: warning: parameter in inline asm statement unused: %2
> "../../../../openmpi-v2.0.1-130-gb3a367d/opal/include/opal/sys/amd64/atomic.h",
>  line 226: warning: parameter in inline asm statement unused: %2
> "../../../../openmpi-v2.0.1-130-gb3a367d/opal/include/opal/sys/amd64/atomic.h",
>  line 247: warning: parameter in inline asm statement unused: %2
> "../../../../openmpi-v2.0.1-130-gb3a367d/opal/include/opal/sys/amd64/atomic.h",
>  line 268: warning: parameter in inline asm statement unused: %2
> "../../../../openmpi-v2.0.1-130-gb3a367d/ompi/mpi/cxx/cxx_glue.h", line 50: 
> identifier redeclared: ompi_errhandler_cxx_dispatch_fn_t
>   current : function(pointer to void, pointer to int, pointer to const 
> char, pointer to function(..) returning void) returning void
>   previous: function(pointer to void, pointer to int, pointer to const 
> char, pointer to function(..) returning void) returning void : 
> "../../../../openmpi-v2.0.1-130-gb3a367d/ompi/errhandler/errhandler.h", line 
> 97
> cc: acomp failed for 
> ../../../../openmpi-v2.0.1-130-gb3a367d/ompi/mpi/cxx/cxx_glue.c
> Makefile:1832: recipe for target 'cxx_glue.lo' failed
> make[2]: *** [cxx_glue.lo] Error 1
> make[2]: Leaving directory 
> '/export2/src/openmpi-2.0.2/openmpi-v2.0.1-130-gb3a367d-Linux.x86_64.64_cc/ompi/mpi/cxx'
> Makefile:3325: recipe for target 'all-recursive' failed
> make[1]: *** [all-recursive] Error 1
> make[1]: Leaving directory 
> '/export2/src/openmpi-2.0.2/openmpi-v2.0.1-130-gb3a367d-Linux.x86_64.64_cc/ompi'
> Makefile:1801: recipe for target 'all-recursive' failed
> make: *** [all-recursive] Error 1
> 
> 
> I would be grateful, if somebody can fix the problem. Thank you
> very much for any help in advance.
> 
> 
> Kind regards
> 
> Siegmar
> ___
> users mailing list
> users@lists.open-mpi.org
> https://rfd.newmexicoconsortium.org/mailman/listinfo/users


-- 
Jeff Squyres
jsquy...@cisco.com
For corporate legal information go to: 
http://www.cisco.com/web/about/doing_business/legal/cri/

___
users mailing list
users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

Re: [OMPI users] OMPI users] Fortran and MPI-3 shared memory

[Jeff Hammond]

Yes, I tried -O0 and -O3.  But VOLATILE is going to thwart a wide range of
optimizations that would break this code.

Jeff

On Thu, Oct 27, 2016 at 2:19 AM, Gilles Gouaillardet <
gilles.gouaillar...@gmail.com> wrote:

> Jeff,
>
> Out of curiosity, did you compile the Fortran test program with -O0 ?
>
> Cheers,
>
> Gilles
>
> Tom Rosmond  wrote:
> Jeff,
>
> Thanks for looking at this.  I know it isn't specific to Open-MPI, but it
> is a frustrating issue vis-a-vis MPI and Fortran.  There are many very
> large MPI applications around the world written in Fortran that could
> benefit greatly from this MPI-3 capability.  My own background is in
> numerical weather prediction, and I know it would be welcome in that
> community.  Someone knowledgeable in both C and Fortran should be able to
> get to bottom of it.
>
> T. Rosmond
>
>
>
> On 10/25/2016 03:05 PM, Jeff Hammond wrote:
>
> I can reproduce this with Intel 17 and MPICH on Mac OSX so it's not an
> Open-MPI issue.  I added VOLATILE to the shared memory objects to prevent
> Fortran compiler optimizations as well as a bunch of MPI_Win_sync calls
> (after replacing fence with lock_all/unlock_all), but neither changed the
> outcome.
>
> While I claim a modest understanding of MPI-3 RMA and Fortran 2008,
> unfortunately, I have never figured out how to use MPI-3 shared memory from
> Fortran, which is especially unfortunate since it seems to be a fantastic
> source of frustration to both real users such as yourself and MPI+Fortran
> standard experts (Rolf).
>
> Sorry for the unsatisfying response, but my suspicion is that this is a
> program correctness issue.  I can't point to any error, but I've ruled out
> the obvious alternatives.
>
> Jeff
>
> On Tue, Oct 25, 2016 at 11:29 AM, Tom Rosmond 
> wrote:
>
>> All:
>>
>> I am trying to understand the use of the shared memory features of MPI-3
>> that allow direct sharing of the memory space of on-node processes.
>> Attached are 2 small test programs, one written in C (testmpi3.c), the
>> other F95 (testmpi3.f90) .  They are solving the identical 'halo' exchange
>> problem.  'testmpi3.c' is a simplified version of an example program from a
>> presentation by Mark Lubin of Intel.  I wrote 'testmpi3.f90' to mimic the C
>> version.
>>
>>  Also attached are 2 text files of the compile, execution, and output of
>> the respective programs:
>>
>> CC_testmpi3.txt
>> F95_testmpi3.txt
>>
>> Note: All 4 files are contained in the attached 'testmpi3.tar.gz'.
>>
>> Comparing the outputs of each version, it is clear that the shared memory
>> copies in 'testmpi3.c' are working correctly, but not in 'testmpi3.f90'.
>> As far as I can tell, the 2 programs are equivalent up to line 134 of
>> 'testmpi3.c' and lines 97-101 of 'testmpi3.f90'. I thought the calls to
>> 'c_f_pointer' would produce Fortran pointers that would access the correct
>> shared memory addresses as the C-pointers do in 'testmpi3.c', but clearly
>> that isn't happening. Can anyone explain why not, and what is needed to
>> make this happen. Any suggestions are welcome.
>>
>> My environment:
>>  Scientific Linux 6.8
>>  INTEL FORTRAN and ICC version 15.0.2.164
>>  OPEN-MPI 2.0.1
>>
>>
>> T. Rosmond
>>
>> ___
>> users mailing list
>> users@lists.open-mpi.org
>> https://rfd.newmexicoconsortium.org/mailman/listinfo/users
>>
>
>
>
> --
> Jeff Hammond
> jeff.scie...@gmail.com
> http://jeffhammond.github.io/
>
>
> ___
> users mailing 
> listus...@lists.open-mpi.orghttps://rfd.newmexicoconsortium.org/mailman/listinfo/users
>
>
>
> ___
> users mailing list
> users@lists.open-mpi.org
> https://rfd.newmexicoconsortium.org/mailman/listinfo/users
>



-- 
Jeff Hammond
jeff.scie...@gmail.com
http://jeffhammond.github.io/
___
users mailing list
users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

Re: [OMPI users] redeclared identifier for openmpi-v2.0.1-130-gb3a367d witj Sun C on Linux

[Gilles Gouaillardet]

Siegmar,

The fix is in the pipe.
Meanwhile, you can download it at 
https://github.com/open-mpi/ompi/pull/2295.patch

Cheers,

Gilles

Siegmar Gross  wrote:
>Hi,
>
>I tried to install openmpi-v2.0.1-130-gb3a367d on my "SUSE Linux
>Enterprise Server 12.1 (x86_64)" with Sun C 5.14 beta. Unfortunately,
>I get the following error. I was able to build it with gcc-6.2.0.
>
>
>loki openmpi-v2.0.1-130-gb3a367d-Linux.x86_64.64_cc 124 tail -18 
>log.make.Linux.x86_64.64_cc
>   CC   cxx_glue.lo
>"../../../../openmpi-v2.0.1-130-gb3a367d/opal/include/opal/sys/amd64/atomic.h",
> 
>line 159: warning: parameter in inline asm statement unused: %3
>"../../../../openmpi-v2.0.1-130-gb3a367d/opal/include/opal/sys/amd64/atomic.h",
> 
>line 205: warning: parameter in inline asm statement unused: %2
>"../../../../openmpi-v2.0.1-130-gb3a367d/opal/include/opal/sys/amd64/atomic.h",
> 
>line 226: warning: parameter in inline asm statement unused: %2
>"../../../../openmpi-v2.0.1-130-gb3a367d/opal/include/opal/sys/amd64/atomic.h",
> 
>line 247: warning: parameter in inline asm statement unused: %2
>"../../../../openmpi-v2.0.1-130-gb3a367d/opal/include/opal/sys/amd64/atomic.h",
> 
>line 268: warning: parameter in inline asm statement unused: %2
>"../../../../openmpi-v2.0.1-130-gb3a367d/ompi/mpi/cxx/cxx_glue.h", line 50: 
>identifier redeclared: ompi_errhandler_cxx_dispatch_fn_t
>   current : function(pointer to void, pointer to int, pointer to const 
> char, 
>pointer to function(..) returning void) returning void
>   previous: function(pointer to void, pointer to int, pointer to const 
> char, 
>pointer to function(..) returning void) returning void : 
>"../../../../openmpi-v2.0.1-130-gb3a367d/ompi/errhandler/errhandler.h", line 97
>cc: acomp failed for 
>../../../../openmpi-v2.0.1-130-gb3a367d/ompi/mpi/cxx/cxx_glue.c
>Makefile:1832: recipe for target 'cxx_glue.lo' failed
>make[2]: *** [cxx_glue.lo] Error 1
>make[2]: Leaving directory 
>'/export2/src/openmpi-2.0.2/openmpi-v2.0.1-130-gb3a367d-Linux.x86_64.64_cc/ompi/mpi/cxx'
>Makefile:3325: recipe for target 'all-recursive' failed
>make[1]: *** [all-recursive] Error 1
>make[1]: Leaving directory 
>'/export2/src/openmpi-2.0.2/openmpi-v2.0.1-130-gb3a367d-Linux.x86_64.64_cc/ompi'
>Makefile:1801: recipe for target 'all-recursive' failed
>make: *** [all-recursive] Error 1
>
>
>I would be grateful, if somebody can fix the problem. Thank you
>very much for any help in advance.
>
>
>Kind regards
>
>Siegmar
>___
>users mailing list
>users@lists.open-mpi.org
>https://rfd.newmexicoconsortium.org/mailman/listinfo/users
___
users mailing list
users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

[OMPI users] redeclared identifier for openmpi-v2.0.1-130-gb3a367d witj Sun C on Linux

[Siegmar Gross]

Hi,

I tried to install openmpi-v2.0.1-130-gb3a367d on my "SUSE Linux
Enterprise Server 12.1 (x86_64)" with Sun C 5.14 beta. Unfortunately,
I get the following error. I was able to build it with gcc-6.2.0.


loki openmpi-v2.0.1-130-gb3a367d-Linux.x86_64.64_cc 124 tail -18 
log.make.Linux.x86_64.64_cc

  CC   cxx_glue.lo
"../../../../openmpi-v2.0.1-130-gb3a367d/opal/include/opal/sys/amd64/atomic.h", 
line 159: warning: parameter in inline asm statement unused: %3
"../../../../openmpi-v2.0.1-130-gb3a367d/opal/include/opal/sys/amd64/atomic.h", 
line 205: warning: parameter in inline asm statement unused: %2
"../../../../openmpi-v2.0.1-130-gb3a367d/opal/include/opal/sys/amd64/atomic.h", 
line 226: warning: parameter in inline asm statement unused: %2
"../../../../openmpi-v2.0.1-130-gb3a367d/opal/include/opal/sys/amd64/atomic.h", 
line 247: warning: parameter in inline asm statement unused: %2
"../../../../openmpi-v2.0.1-130-gb3a367d/opal/include/opal/sys/amd64/atomic.h", 
line 268: warning: parameter in inline asm statement unused: %2
"../../../../openmpi-v2.0.1-130-gb3a367d/ompi/mpi/cxx/cxx_glue.h", line 50: 
identifier redeclared: ompi_errhandler_cxx_dispatch_fn_t
	current : function(pointer to void, pointer to int, pointer to const char, 
pointer to function(..) returning void) returning void
	previous: function(pointer to void, pointer to int, pointer to const char, 
pointer to function(..) returning void) returning void : 
"../../../../openmpi-v2.0.1-130-gb3a367d/ompi/errhandler/errhandler.h", line 97
cc: acomp failed for 
../../../../openmpi-v2.0.1-130-gb3a367d/ompi/mpi/cxx/cxx_glue.c

Makefile:1832: recipe for target 'cxx_glue.lo' failed
make[2]: *** [cxx_glue.lo] Error 1
make[2]: Leaving directory 
'/export2/src/openmpi-2.0.2/openmpi-v2.0.1-130-gb3a367d-Linux.x86_64.64_cc/ompi/mpi/cxx'

Makefile:3325: recipe for target 'all-recursive' failed
make[1]: *** [all-recursive] Error 1
make[1]: Leaving directory 
'/export2/src/openmpi-2.0.2/openmpi-v2.0.1-130-gb3a367d-Linux.x86_64.64_cc/ompi'

Makefile:1801: recipe for target 'all-recursive' failed
make: *** [all-recursive] Error 1


I would be grateful, if somebody can fix the problem. Thank you
very much for any help in advance.


Kind regards

Siegmar
___
users mailing list
users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

Re: [OMPI users] OMPI users] Fortran and MPI-3 shared memory

[Gilles Gouaillardet]

Jeff,

Out of curiosity, did you compile the Fortran test program with -O0 ?

Cheers,

Gilles

Tom Rosmond  wrote:
>Jeff,
>
>Thanks for looking at this.  I know it isn't specific to Open-MPI, but it is a 
>frustrating issue vis-a-vis MPI and Fortran.  There are many very large MPI 
>applications around the world written in Fortran that could benefit greatly 
>from this MPI-3 capability.  My own background is in numerical weather 
>prediction, and I know it would be welcome in that community.  Someone 
>knowledgeable in both C and Fortran should be able to get to bottom of it.
>
>T. Rosmond
>
>
>
>On 10/25/2016 03:05 PM, Jeff Hammond wrote:
>
>I can reproduce this with Intel 17 and MPICH on Mac OSX so it's not an 
>Open-MPI issue.  I added VOLATILE to the shared memory objects to prevent 
>Fortran compiler optimizations as well as a bunch of MPI_Win_sync calls (after 
>replacing fence with lock_all/unlock_all), but neither changed the outcome. 
>
>
>While I claim a modest understanding of MPI-3 RMA and Fortran 2008, 
>unfortunately, I have never figured out how to use MPI-3 shared memory from 
>Fortran, which is especially unfortunate since it seems to be a fantastic 
>source of frustration to both real users such as yourself and MPI+Fortran 
>standard experts (Rolf).
>
>
>Sorry for the unsatisfying response, but my suspicion is that this is a 
>program correctness issue.  I can't point to any error, but I've ruled out the 
>obvious alternatives.
>
>
>Jeff
>
>
>On Tue, Oct 25, 2016 at 11:29 AM, Tom Rosmond  wrote:
>
>All:
>
>I am trying to understand the use of the shared memory features of MPI-3 that 
>allow direct sharing of the memory space of on-node processes.  Attached are 2 
>small test programs, one written in C (testmpi3.c), the other F95 
>(testmpi3.f90) .  They are solving the identical 'halo' exchange problem.  
>'testmpi3.c' is a simplified version of an example program from a presentation 
>by Mark Lubin of Intel.  I wrote 'testmpi3.f90' to mimic the C version.
>
> Also attached are 2 text files of the compile, execution, and output of the 
>respective programs:
>
>CC_testmpi3.txt
>F95_testmpi3.txt
>
>Note: All 4 files are contained in the attached 'testmpi3.tar.gz'.
>
>Comparing the outputs of each version, it is clear that the shared memory 
>copies in 'testmpi3.c' are working correctly, but not in 'testmpi3.f90'.  As 
>far as I can tell, the 2 programs are equivalent up to line 134 of 
>'testmpi3.c' and lines 97-101 of 'testmpi3.f90'. I thought the calls to 
>'c_f_pointer' would produce Fortran pointers that would access the correct 
>shared memory addresses as the C-pointers do in 'testmpi3.c', but clearly that 
>isn't happening. Can anyone explain why not, and what is needed to make this 
>happen. Any suggestions are welcome.
>
>My environment:
> Scientific Linux 6.8
> INTEL FORTRAN and ICC version 15.0.2.164
> OPEN-MPI 2.0.1
>
>
>T. Rosmond
>
>___
>users mailing list
>users@lists.open-mpi.org
>https://rfd.newmexicoconsortium.org/mailman/listinfo/users
>
>
>
>
>-- 
>
>Jeff Hammond
>jeff.scie...@gmail.com
>http://jeffhammond.github.io/
>
>
>
>___ users mailing list 
>users@lists.open-mpi.org 
>https://rfd.newmexicoconsortium.org/mailman/listinfo/users 
>
>
___
users mailing list
users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

Re: [OMPI users] Fortran and MPI-3 shared memory

[Bader, Reinhold]

> Rationale. The C bindings of MPI_ALLOC_MEM and MPI_FREE_MEM are similar
> to the bindings for the malloc and free C library calls: a call to
> MPI_Alloc_mem(: : :, ) should be paired with a call to 
> MPI_Free_mem(base) (one
> less level of indirection). Both arguments are declared to be of same type
> void* so as to facilitate type casting. The Fortran binding is consistent 
> with the C bindings: the Fortran MPI_ALLOC_MEM call returns in baseptr the
> TYPE(C_PTR) pointer or the (integer valued) address of the allocated memory. 
> The base argument of MPI_FREE_MEM is a choice argument, which passes (a 
> reference
> to) the variable
> stored at that location. (End of rationale.)


Understood - this makes clear the MPI interface convention for the C side. 
However, the wording for the Fortran bindings is at best misleading.
If full consistency with the C interface were desired, the baseptr argument 
would need to be declared

TYPE(C_PTR), VALUE :: baseptr

and then users would be required to call the procedure via

CALL MPI_XXX( ..., c_loc(baseptr), ...)

With the signatures as currently defined, I stand by my opinion that the 
internal wrappers need to resolve the * vs ** issue.
See Fortran 2008, 15.3.7 para 2, item (5) (semantics of a dummy argument 
*without* the VALUE attribute).


> at first glance, i also thought type should have been char ** instead of char 
> *, but it is not a typo nor a mistake


> Cheers,


> Gilles


--
Dr. Reinhold Bader
Leibniz Supercomputing Centre (http://www.lrz.de) / HPC Systems and Services
Tel.: 0049 89 35831 8825  Fax: 8625

___
users mailing list
users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

Re: [OMPI users] Fortran and MPI-3 shared memory

[Gilles Gouaillardet]

From the MPI 3.1 standard (page 338)



Rationale. The C bindings of MPI_ALLOC_MEM and MPI_FREE_MEM are similar
to the bindings for the malloc and free C library calls: a call to
MPI_Alloc_mem(: : :, ) should be paired with a call to 
MPI_Free_mem(base) (one

less level of indirection). Both arguments are declared to be of same type
void* so as to facilitate type casting. The Fortran binding is 
consistent with the C
bindings: the Fortran MPI_ALLOC_MEM call returns in baseptr the 
TYPE(C_PTR)
pointer or the (integer valued) address of the allocated memory. The 
base argument
of MPI_FREE_MEM is a choice argument, which passes (a reference to) 
the variable

stored at that location. (End of rationale.)



at first glance, i also thought type should have been char ** instead of 
char *, but it is not a typo nor a mistake



Cheers,


Gilles


On 10/27/2016 5:13 PM, Bader, Reinhold wrote:


I've had a look at the OpenMPI 1.10.3 sources, and the trouble appears 
to me to be that the MPI wrappers declare


the argument

  TYPE(C_PTR), INTENT(OUT) ::  baseptr

inside the BIND(C) interface on the Fortran side (for OpenMPI this 
would, for example be ompi_win_allocate_f), and


the C implementation has "char *baseptr" as parameter. This is 
incorrect, it needs to be


char **baseptr

I believe this actually points to a design weakness of the C-side 
interface. For establishment of a C pointer, usually


functions that return such a pointer are defined (e.g. malloc()). If a 
formal parameter is desired, this should be


a pointer-to-pointer entity, i.e. it would have been better to define 
the MPI_Win_allocate prototype as


int MPI_Win_allocate(MPI_Aint size, int disp_unit, MPI_Info info, 
MPI_comm comm, *void **baseptr*, MPI_Win win);


Note that the C example pretends that this is the case – it passes the 
address of the pointer to


MPI_Win_allocate, not the pointer itself.

Cheers

--

Dr. Reinhold Bader

Leibniz Supercomputing Centre (http://www.lrz.de) / HPC Systems and 
Services


Tel.: 0049 89 35831 8825  Fax: 8625



___
users mailing list
users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/users


___
users mailing list
users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

Re: [OMPI users] Fortran and MPI-3 shared memory

[Bader, Reinhold]

I've had a look at the OpenMPI 1.10.3 sources, and the trouble appears to me to 
be that the MPI wrappers declare
the argument

  TYPE(C_PTR), INTENT(OUT) ::  baseptr

inside the BIND(C) interface on the Fortran side (for OpenMPI this would, for 
example be ompi_win_allocate_f), and
the C implementation has "char *baseptr" as parameter. This is incorrect, it 
needs to be

char **baseptr

I believe this actually points to a design weakness of the C-side interface. 
For establishment of a C pointer, usually
functions that return such a pointer are defined (e.g. malloc()). If a formal 
parameter is desired, this should be
a pointer-to-pointer entity, i.e. it would have been better to define the 
MPI_Win_allocate prototype as

int MPI_Win_allocate(MPI_Aint size, int disp_unit, MPI_Info info, MPI_comm 
comm, void **baseptr, MPI_Win win);

Note that the C example pretends that this is the case - it passes the address 
of the pointer to
MPI_Win_allocate, not the pointer itself.

Cheers


--
Dr. Reinhold Bader
Leibniz Supercomputing Centre (http://www.lrz.de) / HPC Systems and Services
Tel.: 0049 89 35831 8825  Fax: 8625

___
users mailing list
users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/users