Re: [OMPI users] shared memory zero size segment

2016-02-11 Thread Jeff Hammond 
No clarification necessary. Standard is not user guide. Semantics are clear
from what is defined. Users who don't like the interface can write a
library that does what they want.

Jeff

On Thursday, February 11, 2016, Nathan Hjelm  wrote:

>
> I should also say that I think this is something that may be worth
> clarifying in the standard. Either semantic is fine with me but there is
> no reason to change the behavior if it does not violate the standard.
>
> -Nathan
>
> On Thu, Feb 11, 2016 at 01:35:28PM -0700, Nathan Hjelm wrote:
> >
> > Jeff probably ran with MPICH. Open MPI's are consistent with our choice
> > of definition for size=0:
> >
> > query: me=1, them=0, size=0, disp=1, base=0x0
> > query: me=1, them=1, size=4, disp=1, base=0x1097e30f8
> > query: me=1, them=2, size=4, disp=1, base=0x1097e30fc
> > query: me=1, them=3, size=4, disp=1, base=0x1097e3100
> > query: me=2, them=0, size=0, disp=1, base=0x0
> > query: me=2, them=1, size=4, disp=1, base=0x109fe10f8
> > query: me=2, them=2, size=4, disp=1, base=0x109fe10fc
> > query: me=2, them=3, size=4, disp=1, base=0x109fe1100
> > query: me=2, them=PROC_NULL, size=4, disp=1, base=0x109fe10f8
> > query: me=3, them=0, size=0, disp=1, base=0x0
> > query: me=3, them=1, size=4, disp=1, base=0x1088f30f8
> > query: me=3, them=2, size=4, disp=1, base=0x1088f30fc
> > query: me=3, them=3, size=4, disp=1, base=0x1088f3100
> > query: me=3, them=PROC_NULL, size=4, disp=1, base=0x1088f30f8
> > query: me=0, them=0, size=0, disp=1, base=0x0
> > query: me=0, them=1, size=4, disp=1, base=0x105b890f8
> > query: me=0, them=2, size=4, disp=1, base=0x105b890fc
> > query: me=0, them=3, size=4, disp=1, base=0x105b89100
> > query: me=0, them=PROC_NULL, size=4, disp=1, base=0x105b890f8
> > query: me=1, them=PROC_NULL, size=4, disp=1, base=0x1097e30f8
> >
> > To be portable only use MPI_Win_shared_query and do not rely on the
> > return value of base if you pass size = 0.
> >
> > -Nathan
> >
> > On Thu, Feb 11, 2016 at 08:23:16PM +, Peter Wind wrote:
> > >Thanks Jeff, that was an interesting result. The pointers are here
> well
> > >defined, also for the zero size segment.
> > >However I can't reproduce your output. I still get null pointers
> (output
> > >below).
> > >(I tried both 1.8.5 and 1.10.2 versions)
> > >What could be the difference?
> > >Peter
> > >mpirun -np 4 a.out
> > >query: me=0, them=0, size=0, disp=1, base=(nil)
> > >query: me=0, them=1, size=4, disp=1, base=0x2aee280030d0
> > >query: me=0, them=2, size=4, disp=1, base=0x2aee280030d4
> > >query: me=0, them=3, size=4, disp=1, base=0x2aee280030d8
> > >query: me=0, them=PROC_NULL, size=4, disp=1, base=0x2aee280030d0
> > >query: me=1, them=0, size=0, disp=1, base=(nil)
> > >query: me=1, them=1, size=4, disp=1, base=0x2aabbb9ce0d0
> > >query: me=1, them=2, size=4, disp=1, base=0x2aabbb9ce0d4
> > >query: me=1, them=3, size=4, disp=1, base=0x2aabbb9ce0d8
> > >query: me=1, them=PROC_NULL, size=4, disp=1, base=0x2aabbb9ce0d0
> > >query: me=2, them=0, size=0, disp=1, base=(nil)
> > >query: me=2, them=1, size=4, disp=1, base=0x2b1579dd40d0
> > >query: me=2, them=2, size=4, disp=1, base=0x2b1579dd40d4
> > >query: me=2, them=3, size=4, disp=1, base=0x2b1579dd40d8
> > >query: me=2, them=PROC_NULL, size=4, disp=1, base=0x2b1579dd40d0
> > >query: me=3, them=0, size=0, disp=1, base=(nil)
> > >query: me=3, them=1, size=4, disp=1, base=0x2ac8d2c350d0
> > >query: me=3, them=2, size=4, disp=1, base=0x2ac8d2c350d4
> > >query: me=3, them=3, size=4, disp=1, base=0x2ac8d2c350d8
> > >query: me=3, them=PROC_NULL, size=4, disp=1, base=0x2ac8d2c350d0
> > >
> > >
> --
> > >
> > >  See attached.  Output below.  Note that the base you get for
> ranks 0 and
> > >  1 is the same, so you need to use the fact that size=0 at rank=0
> to know
> > >  not to dereference that pointer and expect to be writing into
> rank 0's
> > >  memory, since you will write into rank 1's.
> > >  I would probably add "if (size==0) base=NULL;" for good measure.
> > >  Jeff
> > >
> > >  $ mpirun -n 4 ./a.out
> > >
> > >  query: me=0, them=0, size=0, disp=1, base=0x10bd64000
> > >
> > >  query: me=0, them=1, size=4, disp=1, base=0x10bd64000
> > >
> > >  query: me=0, them=2, size=4, disp=1, base=0x10bd64004
> > >
> > >  query: me=0, them=3, size=4, disp=1, base=0x10bd64008
> > >
> > >  query: me=0, them=PROC_NULL, size=4, disp=1, base=0x10bd64000
> > >
> > >  query: me=1, them=0, size=0, disp=1, base=0x102d3b000
> > >
> > >  query: me=1, them=1, size=4, disp=1, base=0x102d3b000
> > >
> > >  query: me=1, them=2, size=4, disp=1, base=0x102d3b004
> > >
> > >  query: me=1, them=3, size=4, disp=1, base=0x102d3b008
> > >
> > >  query: me=1, them=PROC_NULL, size=4, disp=1, base=0x102d3b000
> > >
> > >  query:

Re: [OMPI users] shared memory zero size segment

2016-02-11 Thread Jeff Hammond 
Indeed, I ran with MPICH. But I like OpenMPI's choice better here, which is
why I said that I would explicitly set the pointer to bull when size is
zero.

Jeff

On Thursday, February 11, 2016, Nathan Hjelm  wrote:

>
> Jeff probably ran with MPICH. Open MPI's are consistent with our choice
> of definition for size=0:
>
> query: me=1, them=0, size=0, disp=1, base=0x0
> query: me=1, them=1, size=4, disp=1, base=0x1097e30f8
> query: me=1, them=2, size=4, disp=1, base=0x1097e30fc
> query: me=1, them=3, size=4, disp=1, base=0x1097e3100
> query: me=2, them=0, size=0, disp=1, base=0x0
> query: me=2, them=1, size=4, disp=1, base=0x109fe10f8
> query: me=2, them=2, size=4, disp=1, base=0x109fe10fc
> query: me=2, them=3, size=4, disp=1, base=0x109fe1100
> query: me=2, them=PROC_NULL, size=4, disp=1, base=0x109fe10f8
> query: me=3, them=0, size=0, disp=1, base=0x0
> query: me=3, them=1, size=4, disp=1, base=0x1088f30f8
> query: me=3, them=2, size=4, disp=1, base=0x1088f30fc
> query: me=3, them=3, size=4, disp=1, base=0x1088f3100
> query: me=3, them=PROC_NULL, size=4, disp=1, base=0x1088f30f8
> query: me=0, them=0, size=0, disp=1, base=0x0
> query: me=0, them=1, size=4, disp=1, base=0x105b890f8
> query: me=0, them=2, size=4, disp=1, base=0x105b890fc
> query: me=0, them=3, size=4, disp=1, base=0x105b89100
> query: me=0, them=PROC_NULL, size=4, disp=1, base=0x105b890f8
> query: me=1, them=PROC_NULL, size=4, disp=1, base=0x1097e30f8
>
> To be portable only use MPI_Win_shared_query and do not rely on the
> return value of base if you pass size = 0.
>
> -Nathan
>
> On Thu, Feb 11, 2016 at 08:23:16PM +, Peter Wind wrote:
> >Thanks Jeff, that was an interesting result. The pointers are here
> well
> >defined, also for the zero size segment.
> >However I can't reproduce your output. I still get null pointers
> (output
> >below).
> >(I tried both 1.8.5 and 1.10.2 versions)
> >What could be the difference?
> >Peter
> >mpirun -np 4 a.out
> >query: me=0, them=0, size=0, disp=1, base=(nil)
> >query: me=0, them=1, size=4, disp=1, base=0x2aee280030d0
> >query: me=0, them=2, size=4, disp=1, base=0x2aee280030d4
> >query: me=0, them=3, size=4, disp=1, base=0x2aee280030d8
> >query: me=0, them=PROC_NULL, size=4, disp=1, base=0x2aee280030d0
> >query: me=1, them=0, size=0, disp=1, base=(nil)
> >query: me=1, them=1, size=4, disp=1, base=0x2aabbb9ce0d0
> >query: me=1, them=2, size=4, disp=1, base=0x2aabbb9ce0d4
> >query: me=1, them=3, size=4, disp=1, base=0x2aabbb9ce0d8
> >query: me=1, them=PROC_NULL, size=4, disp=1, base=0x2aabbb9ce0d0
> >query: me=2, them=0, size=0, disp=1, base=(nil)
> >query: me=2, them=1, size=4, disp=1, base=0x2b1579dd40d0
> >query: me=2, them=2, size=4, disp=1, base=0x2b1579dd40d4
> >query: me=2, them=3, size=4, disp=1, base=0x2b1579dd40d8
> >query: me=2, them=PROC_NULL, size=4, disp=1, base=0x2b1579dd40d0
> >query: me=3, them=0, size=0, disp=1, base=(nil)
> >query: me=3, them=1, size=4, disp=1, base=0x2ac8d2c350d0
> >query: me=3, them=2, size=4, disp=1, base=0x2ac8d2c350d4
> >query: me=3, them=3, size=4, disp=1, base=0x2ac8d2c350d8
> >query: me=3, them=PROC_NULL, size=4, disp=1, base=0x2ac8d2c350d0
> >
> >
> --
> >
> >  See attached.  Output below.  Note that the base you get for ranks
> 0 and
> >  1 is the same, so you need to use the fact that size=0 at rank=0 to
> know
> >  not to dereference that pointer and expect to be writing into rank
> 0's
> >  memory, since you will write into rank 1's.
> >  I would probably add "if (size==0) base=NULL;" for good measure.
> >  Jeff
> >
> >  $ mpirun -n 4 ./a.out
> >
> >  query: me=0, them=0, size=0, disp=1, base=0x10bd64000
> >
> >  query: me=0, them=1, size=4, disp=1, base=0x10bd64000
> >
> >  query: me=0, them=2, size=4, disp=1, base=0x10bd64004
> >
> >  query: me=0, them=3, size=4, disp=1, base=0x10bd64008
> >
> >  query: me=0, them=PROC_NULL, size=4, disp=1, base=0x10bd64000
> >
> >  query: me=1, them=0, size=0, disp=1, base=0x102d3b000
> >
> >  query: me=1, them=1, size=4, disp=1, base=0x102d3b000
> >
> >  query: me=1, them=2, size=4, disp=1, base=0x102d3b004
> >
> >  query: me=1, them=3, size=4, disp=1, base=0x102d3b008
> >
> >  query: me=1, them=PROC_NULL, size=4, disp=1, base=0x102d3b000
> >
> >  query: me=2, them=0, size=0, disp=1, base=0x10aac1000
> >
> >  query: me=2, them=1, size=4, disp=1, base=0x10aac1000
> >
> >  query: me=2, them=2, size=4, disp=1, base=0x10aac1004
> >
> >  query: me=2, them=3, size=4, disp=1, base=0x10aac1008
> >
> >  query: me=2, them=PROC_NULL, size=4, disp=1, base=0x10aac1000
> >
> >  query: me=3, them=0, size=0, disp=1, base=0x100fa2000
> >
> >  query: me=3, them=1, size=4, disp=1, base=0x100fa2000
> >
> >  query: me=3, them=2, size=4,

Re: [OMPI users] shared memory zero size segment

2016-02-11 Thread Nathan Hjelm 

I should also say that I think this is something that may be worth
clarifying in the standard. Either semantic is fine with me but there is
no reason to change the behavior if it does not violate the standard.

-Nathan

On Thu, Feb 11, 2016 at 01:35:28PM -0700, Nathan Hjelm wrote:
> 
> Jeff probably ran with MPICH. Open MPI's are consistent with our choice
> of definition for size=0:
> 
> query: me=1, them=0, size=0, disp=1, base=0x0
> query: me=1, them=1, size=4, disp=1, base=0x1097e30f8
> query: me=1, them=2, size=4, disp=1, base=0x1097e30fc
> query: me=1, them=3, size=4, disp=1, base=0x1097e3100
> query: me=2, them=0, size=0, disp=1, base=0x0
> query: me=2, them=1, size=4, disp=1, base=0x109fe10f8
> query: me=2, them=2, size=4, disp=1, base=0x109fe10fc
> query: me=2, them=3, size=4, disp=1, base=0x109fe1100
> query: me=2, them=PROC_NULL, size=4, disp=1, base=0x109fe10f8
> query: me=3, them=0, size=0, disp=1, base=0x0
> query: me=3, them=1, size=4, disp=1, base=0x1088f30f8
> query: me=3, them=2, size=4, disp=1, base=0x1088f30fc
> query: me=3, them=3, size=4, disp=1, base=0x1088f3100
> query: me=3, them=PROC_NULL, size=4, disp=1, base=0x1088f30f8
> query: me=0, them=0, size=0, disp=1, base=0x0
> query: me=0, them=1, size=4, disp=1, base=0x105b890f8
> query: me=0, them=2, size=4, disp=1, base=0x105b890fc
> query: me=0, them=3, size=4, disp=1, base=0x105b89100
> query: me=0, them=PROC_NULL, size=4, disp=1, base=0x105b890f8
> query: me=1, them=PROC_NULL, size=4, disp=1, base=0x1097e30f8
> 
> To be portable only use MPI_Win_shared_query and do not rely on the
> return value of base if you pass size = 0.
> 
> -Nathan
> 
> On Thu, Feb 11, 2016 at 08:23:16PM +, Peter Wind wrote:
> >Thanks Jeff, that was an interesting result. The pointers are here well
> >defined, also for the zero size segment.
> >However I can't reproduce your output. I still get null pointers (output
> >below).
> >(I tried both 1.8.5 and 1.10.2 versions)
> >What could be the difference?
> >Peter
> >mpirun -np 4 a.out
> >query: me=0, them=0, size=0, disp=1, base=(nil)
> >query: me=0, them=1, size=4, disp=1, base=0x2aee280030d0
> >query: me=0, them=2, size=4, disp=1, base=0x2aee280030d4
> >query: me=0, them=3, size=4, disp=1, base=0x2aee280030d8
> >query: me=0, them=PROC_NULL, size=4, disp=1, base=0x2aee280030d0
> >query: me=1, them=0, size=0, disp=1, base=(nil)
> >query: me=1, them=1, size=4, disp=1, base=0x2aabbb9ce0d0
> >query: me=1, them=2, size=4, disp=1, base=0x2aabbb9ce0d4
> >query: me=1, them=3, size=4, disp=1, base=0x2aabbb9ce0d8
> >query: me=1, them=PROC_NULL, size=4, disp=1, base=0x2aabbb9ce0d0
> >query: me=2, them=0, size=0, disp=1, base=(nil)
> >query: me=2, them=1, size=4, disp=1, base=0x2b1579dd40d0
> >query: me=2, them=2, size=4, disp=1, base=0x2b1579dd40d4
> >query: me=2, them=3, size=4, disp=1, base=0x2b1579dd40d8
> >query: me=2, them=PROC_NULL, size=4, disp=1, base=0x2b1579dd40d0
> >query: me=3, them=0, size=0, disp=1, base=(nil)
> >query: me=3, them=1, size=4, disp=1, base=0x2ac8d2c350d0
> >query: me=3, them=2, size=4, disp=1, base=0x2ac8d2c350d4
> >query: me=3, them=3, size=4, disp=1, base=0x2ac8d2c350d8
> >query: me=3, them=PROC_NULL, size=4, disp=1, base=0x2ac8d2c350d0
> > 
> >  --
> > 
> >  See attached.  Output below.  Note that the base you get for ranks 0 
> > and
> >  1 is the same, so you need to use the fact that size=0 at rank=0 to 
> > know
> >  not to dereference that pointer and expect to be writing into rank 0's
> >  memory, since you will write into rank 1's.
> >  I would probably add "if (size==0) base=NULL;" for good measure.
> >  Jeff
> > 
> >  $ mpirun -n 4 ./a.out
> > 
> >  query: me=0, them=0, size=0, disp=1, base=0x10bd64000
> > 
> >  query: me=0, them=1, size=4, disp=1, base=0x10bd64000
> > 
> >  query: me=0, them=2, size=4, disp=1, base=0x10bd64004
> > 
> >  query: me=0, them=3, size=4, disp=1, base=0x10bd64008
> > 
> >  query: me=0, them=PROC_NULL, size=4, disp=1, base=0x10bd64000
> > 
> >  query: me=1, them=0, size=0, disp=1, base=0x102d3b000
> > 
> >  query: me=1, them=1, size=4, disp=1, base=0x102d3b000
> > 
> >  query: me=1, them=2, size=4, disp=1, base=0x102d3b004
> > 
> >  query: me=1, them=3, size=4, disp=1, base=0x102d3b008
> > 
> >  query: me=1, them=PROC_NULL, size=4, disp=1, base=0x102d3b000
> > 
> >  query: me=2, them=0, size=0, disp=1, base=0x10aac1000
> > 
> >  query: me=2, them=1, size=4, disp=1, base=0x10aac1000
> > 
> >  query: me=2, them=2, size=4, disp=1, base=0x10aac1004
> > 
> >  query: me=2, them=3, size=4, disp=1, base=0x10aac1008
> > 
> >  query: me=2, them=PROC_NULL, size=4, disp=1, base=0x10aac1000
> > 
> >  query: me=3, them=0, size=0, disp=1, base=0x100fa2000
> > 
> >  query: me=3, them=1,

Re: [OMPI users] shared memory zero size segment

2016-02-11 Thread Nathan Hjelm 

Jeff probably ran with MPICH. Open MPI's are consistent with our choice
of definition for size=0:

query: me=1, them=0, size=0, disp=1, base=0x0
query: me=1, them=1, size=4, disp=1, base=0x1097e30f8
query: me=1, them=2, size=4, disp=1, base=0x1097e30fc
query: me=1, them=3, size=4, disp=1, base=0x1097e3100
query: me=2, them=0, size=0, disp=1, base=0x0
query: me=2, them=1, size=4, disp=1, base=0x109fe10f8
query: me=2, them=2, size=4, disp=1, base=0x109fe10fc
query: me=2, them=3, size=4, disp=1, base=0x109fe1100
query: me=2, them=PROC_NULL, size=4, disp=1, base=0x109fe10f8
query: me=3, them=0, size=0, disp=1, base=0x0
query: me=3, them=1, size=4, disp=1, base=0x1088f30f8
query: me=3, them=2, size=4, disp=1, base=0x1088f30fc
query: me=3, them=3, size=4, disp=1, base=0x1088f3100
query: me=3, them=PROC_NULL, size=4, disp=1, base=0x1088f30f8
query: me=0, them=0, size=0, disp=1, base=0x0
query: me=0, them=1, size=4, disp=1, base=0x105b890f8
query: me=0, them=2, size=4, disp=1, base=0x105b890fc
query: me=0, them=3, size=4, disp=1, base=0x105b89100
query: me=0, them=PROC_NULL, size=4, disp=1, base=0x105b890f8
query: me=1, them=PROC_NULL, size=4, disp=1, base=0x1097e30f8

To be portable only use MPI_Win_shared_query and do not rely on the
return value of base if you pass size = 0.

-Nathan

On Thu, Feb 11, 2016 at 08:23:16PM +, Peter Wind wrote:
>Thanks Jeff, that was an interesting result. The pointers are here well
>defined, also for the zero size segment.
>However I can't reproduce your output. I still get null pointers (output
>below).
>(I tried both 1.8.5 and 1.10.2 versions)
>What could be the difference?
>Peter
>mpirun -np 4 a.out
>query: me=0, them=0, size=0, disp=1, base=(nil)
>query: me=0, them=1, size=4, disp=1, base=0x2aee280030d0
>query: me=0, them=2, size=4, disp=1, base=0x2aee280030d4
>query: me=0, them=3, size=4, disp=1, base=0x2aee280030d8
>query: me=0, them=PROC_NULL, size=4, disp=1, base=0x2aee280030d0
>query: me=1, them=0, size=0, disp=1, base=(nil)
>query: me=1, them=1, size=4, disp=1, base=0x2aabbb9ce0d0
>query: me=1, them=2, size=4, disp=1, base=0x2aabbb9ce0d4
>query: me=1, them=3, size=4, disp=1, base=0x2aabbb9ce0d8
>query: me=1, them=PROC_NULL, size=4, disp=1, base=0x2aabbb9ce0d0
>query: me=2, them=0, size=0, disp=1, base=(nil)
>query: me=2, them=1, size=4, disp=1, base=0x2b1579dd40d0
>query: me=2, them=2, size=4, disp=1, base=0x2b1579dd40d4
>query: me=2, them=3, size=4, disp=1, base=0x2b1579dd40d8
>query: me=2, them=PROC_NULL, size=4, disp=1, base=0x2b1579dd40d0
>query: me=3, them=0, size=0, disp=1, base=(nil)
>query: me=3, them=1, size=4, disp=1, base=0x2ac8d2c350d0
>query: me=3, them=2, size=4, disp=1, base=0x2ac8d2c350d4
>query: me=3, them=3, size=4, disp=1, base=0x2ac8d2c350d8
>query: me=3, them=PROC_NULL, size=4, disp=1, base=0x2ac8d2c350d0
> 
>  --
> 
>  See attached.  Output below.  Note that the base you get for ranks 0 and
>  1 is the same, so you need to use the fact that size=0 at rank=0 to know
>  not to dereference that pointer and expect to be writing into rank 0's
>  memory, since you will write into rank 1's.
>  I would probably add "if (size==0) base=NULL;" for good measure.
>  Jeff
> 
>  $ mpirun -n 4 ./a.out
> 
>  query: me=0, them=0, size=0, disp=1, base=0x10bd64000
> 
>  query: me=0, them=1, size=4, disp=1, base=0x10bd64000
> 
>  query: me=0, them=2, size=4, disp=1, base=0x10bd64004
> 
>  query: me=0, them=3, size=4, disp=1, base=0x10bd64008
> 
>  query: me=0, them=PROC_NULL, size=4, disp=1, base=0x10bd64000
> 
>  query: me=1, them=0, size=0, disp=1, base=0x102d3b000
> 
>  query: me=1, them=1, size=4, disp=1, base=0x102d3b000
> 
>  query: me=1, them=2, size=4, disp=1, base=0x102d3b004
> 
>  query: me=1, them=3, size=4, disp=1, base=0x102d3b008
> 
>  query: me=1, them=PROC_NULL, size=4, disp=1, base=0x102d3b000
> 
>  query: me=2, them=0, size=0, disp=1, base=0x10aac1000
> 
>  query: me=2, them=1, size=4, disp=1, base=0x10aac1000
> 
>  query: me=2, them=2, size=4, disp=1, base=0x10aac1004
> 
>  query: me=2, them=3, size=4, disp=1, base=0x10aac1008
> 
>  query: me=2, them=PROC_NULL, size=4, disp=1, base=0x10aac1000
> 
>  query: me=3, them=0, size=0, disp=1, base=0x100fa2000
> 
>  query: me=3, them=1, size=4, disp=1, base=0x100fa2000
> 
>  query: me=3, them=2, size=4, disp=1, base=0x100fa2004
> 
>  query: me=3, them=3, size=4, disp=1, base=0x100fa2008
> 
>  query: me=3, them=PROC_NULL, size=4, disp=1, base=0x100fa2000
> 
>  On Thu, Feb 11, 2016 at 8:55 AM, Jeff Hammond 
>  wrote:
> 
>On Thu, Feb 11, 2016 at 8:46 AM, Nathan Hjelm  wrote:
>>
>>
>> On Thu, Feb 11, 2016 at 02:17:40PM +, Peter Wind

Re: [OMPI users] shared memory zero size segment

2016-02-11 Thread Peter Wind 
You can be right semantically. But also the sentence "the first address in the 
memory segment of process i is consecutive with the last address in the memory 
segment of process i - 1" is not easy to interpret correctly for a zero size 
segment.

There may be good reasons not to allocate the pointer for zero size segment. 
What I try to say is, that a new user reading the documentation, will not 
expect this behaviour before trying it out.
Couldn't a small sentence in the documentation, like "the pointer should not be 
used for zero size segments" clarify this?

Peter

- Original Message -
> 
> On Thu, Feb 11, 2016 at 02:17:40PM +, Peter Wind wrote:
> >I would add that the present situation is bound to give problems for
> >some
> >users.
> >It is natural to divide an array in segments, each process treating its
> >own segment, but needing to read adjacent segments too.
> >MPI_Win_allocate_shared seems to be designed for this.
> >This will work fine as long as no segment as size zero. It can also be
> >expected that most testing would be done with all segments larger than
> >zero.
> >The document adding "size = 0 is valid", would also make people
> >confident
> >that it will be consistent for that special case too.
> 
> Nope, that statement says its ok for a rank to specify that the local
> shared memory segment is 0 bytes. Nothing more. The standard
> unfortunately does not define what pointer value is returned for a rank
> that specifies size = 0. Not sure if the RMA working group intentionally
> left that undefine... Anyway, Open MPI does not appear to be out of
> compliance with the standard here.
> 
> To be safe you should use MPI_Win_shared_query as suggested. You can
> pass MPI_PROC_NULL as the rank to get the pointer for the first non-zero
> sized segment in the shared memory window.
> 
> -Nathan
> HPC-5, LANL
> 
> ___
> users mailing list
> us...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> Link to this post:
> http://www.open-mpi.org/community/lists/users/2016/02/28506.php

Re: [OMPI users] shared memory zero size segment

2016-02-11 Thread Peter Wind 
Thanks Jeff, that was an interesting result. The pointers are here well 
defined, also for the zero size segment. 
However I can't reproduce your output. I still get null pointers (output 
below). 
(I tried both 1.8.5 and 1.10.2 versions) 
What could be the difference? 

Peter 

mpirun -np 4 a.out 
query: me=0, them=0, size=0, disp=1, base=(nil) 
query: me=0, them=1, size=4, disp=1, base=0x2aee280030d0 
query: me=0, them=2, size=4, disp=1, base=0x2aee280030d4 
query: me=0, them=3, size=4, disp=1, base=0x2aee280030d8 
query: me=0, them=PROC_NULL, size=4, disp=1, base=0x2aee280030d0 
query: me=1, them=0, size=0, disp=1, base=(nil) 
query: me=1, them=1, size=4, disp=1, base=0x2aabbb9ce0d0 
query: me=1, them=2, size=4, disp=1, base=0x2aabbb9ce0d4 
query: me=1, them=3, size=4, disp=1, base=0x2aabbb9ce0d8 
query: me=1, them=PROC_NULL, size=4, disp=1, base=0x2aabbb9ce0d0 
query: me=2, them=0, size=0, disp=1, base=(nil) 
query: me=2, them=1, size=4, disp=1, base=0x2b1579dd40d0 
query: me=2, them=2, size=4, disp=1, base=0x2b1579dd40d4 
query: me=2, them=3, size=4, disp=1, base=0x2b1579dd40d8 
query: me=2, them=PROC_NULL, size=4, disp=1, base=0x2b1579dd40d0 
query: me=3, them=0, size=0, disp=1, base=(nil) 
query: me=3, them=1, size=4, disp=1, base=0x2ac8d2c350d0 
query: me=3, them=2, size=4, disp=1, base=0x2ac8d2c350d4 
query: me=3, them=3, size=4, disp=1, base=0x2ac8d2c350d8 
query: me=3, them=PROC_NULL, size=4, disp=1, base=0x2ac8d2c350d0 

- Original Message -

> See attached. Output below. Note that the base you get for ranks 0 and 1 is
> the same, so you need to use the fact that size=0 at rank=0 to know not to
> dereference that pointer and expect to be writing into rank 0's memory,
> since you will write into rank 1's.

> I would probably add "if (size==0) base=NULL;" for good measure.

> Jeff

> $ mpirun -n 4 ./a.out

> query: me=0, them=0, size=0, disp=1, base=0x10bd64000

> query: me=0, them=1, size=4, disp=1, base=0x10bd64000

> query: me=0, them=2, size=4, disp=1, base=0x10bd64004

> query: me=0, them=3, size=4, disp=1, base=0x10bd64008

> query: me=0, them=PROC_NULL, size=4, disp=1, base=0x10bd64000

> query: me=1, them=0, size=0, disp=1, base=0x102d3b000

> query: me=1, them=1, size=4, disp=1, base=0x102d3b000

> query: me=1, them=2, size=4, disp=1, base=0x102d3b004

> query: me=1, them=3, size=4, disp=1, base=0x102d3b008

> query: me=1, them=PROC_NULL, size=4, disp=1, base=0x102d3b000

> query: me=2, them=0, size=0, disp=1, base=0x10aac1000

> query: me=2, them=1, size=4, disp=1, base=0x10aac1000

> query: me=2, them=2, size=4, disp=1, base=0x10aac1004

> query: me=2, them=3, size=4, disp=1, base=0x10aac1008

> query: me=2, them=PROC_NULL, size=4, disp=1, base=0x10aac1000

> query: me=3, them=0, size=0, disp=1, base=0x100fa2000

> query: me=3, them=1, size=4, disp=1, base=0x100fa2000

> query: me=3, them=2, size=4, disp=1, base=0x100fa2004

> query: me=3, them=3, size=4, disp=1, base=0x100fa2008

> query: me=3, them=PROC_NULL, size=4, disp=1, base=0x100fa2000

> On Thu, Feb 11, 2016 at 8:55 AM, Jeff Hammond < jeff.scie...@gmail.com >
> wrote:

> > On Thu, Feb 11, 2016 at 8:46 AM, Nathan Hjelm < hje...@lanl.gov > wrote:
> 
> > >
> 
> > >
> 
> > > On Thu, Feb 11, 2016 at 02:17:40PM +, Peter Wind wrote:
> 
> > > > I would add that the present situation is bound to give problems for
> > > > some
> 
> > > > users.
> 
> > > > It is natural to divide an array in segments, each process treating its
> 
> > > > own segment, but needing to read adjacent segments too.
> 
> > > > MPI_Win_allocate_shared seems to be designed for this.
> 
> > > > This will work fine as long as no segment as size zero. It can also be
> 
> > > > expected that most testing would be done with all segments larger than
> 
> > > > zero.
> 
> > > > The document adding "size = 0 is valid", would also make people
> > > > confident
> 
> > > > that it will be consistent for that special case too.
> 
> > >
> 
> > > Nope, that statement says its ok for a rank to specify that the local
> 
> > > shared memory segment is 0 bytes. Nothing more. The standard
> 
> > > unfortunately does not define what pointer value is returned for a rank
> 
> > > that specifies size = 0. Not sure if the RMA working group intentionally
> 
> > > left that undefine... Anyway, Open MPI does not appear to be out of
> 
> > > compliance with the standard here.
> 
> > >
> 

> > MPI_Alloc_mem doesn't say what happens if you pass size=0 either. The RMA
> > working group intentionally tries to maintain consistency with the rest of
> > the MPI standard whenever possible, so we did not create a new semantic
> > here.
> 

> > MPI_Win_shared_query text includes this:
> 

> > "If all processes in the group attached to the window specified size = 0,
> > then the call returns size = 0 and a baseptr as if MPI_ALLOC_MEM was called
> > with size = 0."
> 

> > >
> 
> > > To be safe you should use MPI_Win_shared_query as suggested. You can
> 
> > > pass MPI_PROC_NULL as the

Re: [OMPI users] shared memory zero size segment

2016-02-11 Thread Jeff Hammond 
See attached.  Output below.  Note that the base you get for ranks 0 and 1
is the same, so you need to use the fact that size=0 at rank=0 to know not
to dereference that pointer and expect to be writing into rank 0's memory,
since you will write into rank 1's.

I would probably add "if (size==0) base=NULL;" for good measure.

Jeff

$ mpirun -n 4 ./a.out

query: me=0, them=0, size=0, disp=1, base=0x10bd64000

query: me=0, them=1, size=4, disp=1, base=0x10bd64000

query: me=0, them=2, size=4, disp=1, base=0x10bd64004

query: me=0, them=3, size=4, disp=1, base=0x10bd64008

query: me=0, them=PROC_NULL, size=4, disp=1, base=0x10bd64000

query: me=1, them=0, size=0, disp=1, base=0x102d3b000

query: me=1, them=1, size=4, disp=1, base=0x102d3b000

query: me=1, them=2, size=4, disp=1, base=0x102d3b004

query: me=1, them=3, size=4, disp=1, base=0x102d3b008

query: me=1, them=PROC_NULL, size=4, disp=1, base=0x102d3b000

query: me=2, them=0, size=0, disp=1, base=0x10aac1000

query: me=2, them=1, size=4, disp=1, base=0x10aac1000

query: me=2, them=2, size=4, disp=1, base=0x10aac1004

query: me=2, them=3, size=4, disp=1, base=0x10aac1008

query: me=2, them=PROC_NULL, size=4, disp=1, base=0x10aac1000

query: me=3, them=0, size=0, disp=1, base=0x100fa2000

query: me=3, them=1, size=4, disp=1, base=0x100fa2000

query: me=3, them=2, size=4, disp=1, base=0x100fa2004

query: me=3, them=3, size=4, disp=1, base=0x100fa2008

query: me=3, them=PROC_NULL, size=4, disp=1, base=0x100fa2000

On Thu, Feb 11, 2016 at 8:55 AM, Jeff Hammond 
wrote:

>
>
> On Thu, Feb 11, 2016 at 8:46 AM, Nathan Hjelm  wrote:
> >
> >
> > On Thu, Feb 11, 2016 at 02:17:40PM +, Peter Wind wrote:
> > >I would add that the present situation is bound to give problems
> for some
> > >users.
> > >It is natural to divide an array in segments, each process treating
> its
> > >own segment, but needing to read adjacent segments too.
> > >MPI_Win_allocate_shared seems to be designed for this.
> > >This will work fine as long as no segment as size zero. It can also
> be
> > >expected that most testing would be done with all segments larger
> than
> > >zero.
> > >The document adding "size = 0 is valid", would also make people
> confident
> > >that it will be consistent for that special case too.
> >
> > Nope, that statement says its ok for a rank to specify that the local
> > shared memory segment is 0 bytes. Nothing more. The standard
> > unfortunately does not define what pointer value is returned for a rank
> > that specifies size = 0. Not sure if the RMA working group intentionally
> > left that undefine... Anyway, Open MPI does not appear to be out of
> > compliance with the standard here.
> >
>
> MPI_Alloc_mem doesn't say what happens if you pass size=0 either.  The RMA
> working group intentionally tries to maintain consistency with the rest of
> the MPI standard whenever possible, so we did not create a new semantic
> here.
>
> MPI_Win_shared_query text includes this:
>
> "If all processes in the group attached to the window specified size = 0,
> then the call returns size = 0 and a baseptr as if MPI_ALLOC_MEM was called
> with size = 0."
>
> >
> > To be safe you should use MPI_Win_shared_query as suggested. You can
> > pass MPI_PROC_NULL as the rank to get the pointer for the first non-zero
> > sized segment in the shared memory window.
>
> Indeed!  I forgot about that.  MPI_Win_shared_query solves this problem
> for the user brilliantly.
>
> Jeff
>
> --
> Jeff Hammond
> jeff.scie...@gmail.com
> http://jeffhammond.github.io/
>



-- 
Jeff Hammond
jeff.scie...@gmail.com
http://jeffhammond.github.io/
#include 
#include 

/* test zero size segment.
 run on at least 3 cpus
 mpirun -np 4 a.out */

int main(int argc, char** argv)
{
MPI_Init(NULL, NULL);

int wsize, wrank;
MPI_Comm_size(MPI_COMM_WORLD, );
MPI_Comm_rank(MPI_COMM_WORLD, );

MPI_Comm ncomm = MPI_COMM_NULL;
MPI_Comm_split_type(MPI_COMM_WORLD, MPI_COMM_TYPE_SHARED, 0, MPI_INFO_NULL, );

MPI_Aint size = (wrank==0) ? 0 : sizeof(int);
MPI_Win win = MPI_WIN_NULL;
int * ptr = NULL;
MPI_Win_allocate_shared(size, 1, MPI_INFO_NULL, MPI_COMM_WORLD, , );

int nsize, nrank;
MPI_Comm_size(MPI_COMM_WORLD, );
MPI_Comm_rank(MPI_COMM_WORLD, );

for (int r=0; r

Re: [OMPI users] shared memory zero size segment

2016-02-11 Thread Jeff Hammond 
On Thu, Feb 11, 2016 at 8:46 AM, Nathan Hjelm  wrote:
>
>
> On Thu, Feb 11, 2016 at 02:17:40PM +, Peter Wind wrote:
> >I would add that the present situation is bound to give problems for
some
> >users.
> >It is natural to divide an array in segments, each process treating
its
> >own segment, but needing to read adjacent segments too.
> >MPI_Win_allocate_shared seems to be designed for this.
> >This will work fine as long as no segment as size zero. It can also
be
> >expected that most testing would be done with all segments larger
than
> >zero.
> >The document adding "size = 0 is valid", would also make people
confident
> >that it will be consistent for that special case too.
>
> Nope, that statement says its ok for a rank to specify that the local
> shared memory segment is 0 bytes. Nothing more. The standard
> unfortunately does not define what pointer value is returned for a rank
> that specifies size = 0. Not sure if the RMA working group intentionally
> left that undefine... Anyway, Open MPI does not appear to be out of
> compliance with the standard here.
>

MPI_Alloc_mem doesn't say what happens if you pass size=0 either.  The RMA
working group intentionally tries to maintain consistency with the rest of
the MPI standard whenever possible, so we did not create a new semantic
here.

MPI_Win_shared_query text includes this:

"If all processes in the group attached to the window specified size = 0,
then the call returns size = 0 and a baseptr as if MPI_ALLOC_MEM was called
with size = 0."

>
> To be safe you should use MPI_Win_shared_query as suggested. You can
> pass MPI_PROC_NULL as the rank to get the pointer for the first non-zero
> sized segment in the shared memory window.

Indeed!  I forgot about that.  MPI_Win_shared_query solves this problem for
the user brilliantly.

Jeff

--
Jeff Hammond
jeff.scie...@gmail.com
http://jeffhammond.github.io/

Re: [OMPI users] shared memory zero size segment

2016-02-11 Thread Nathan Hjelm 

On Thu, Feb 11, 2016 at 02:17:40PM +, Peter Wind wrote:
>I would add that the present situation is bound to give problems for some
>users.
>It is natural to divide an array in segments, each process treating its
>own segment, but needing to read adjacent segments too.
>MPI_Win_allocate_shared seems to be designed for this.
>This will work fine as long as no segment as size zero. It can also be
>expected that most testing would be done with all segments larger than
>zero.
>The document adding "size = 0 is valid", would also make people confident
>that it will be consistent for that special case too.

Nope, that statement says its ok for a rank to specify that the local
shared memory segment is 0 bytes. Nothing more. The standard
unfortunately does not define what pointer value is returned for a rank
that specifies size = 0. Not sure if the RMA working group intentionally
left that undefine... Anyway, Open MPI does not appear to be out of
compliance with the standard here.

To be safe you should use MPI_Win_shared_query as suggested. You can
pass MPI_PROC_NULL as the rank to get the pointer for the first non-zero
sized segment in the shared memory window.

-Nathan
HPC-5, LANL


pgp4wih319kzq.pgp
Description: PGP signature

Re: [OMPI users] shared memory zero size segment

2016-02-11 Thread Peter Wind 
I would add that the present situation is bound to give problems for some 
users. 

It is natural to divide an array in segments, each process treating its own 
segment, but needing to read adjacent segments too. 
MPI_Win_allocate_shared seems to be designed for this. 
This will work fine as long as no segment as size zero. It can also be expected 
that most testing would be done with all segments larger than zero. 
The document adding "size = 0 is valid", would also make people confident that 
it will be consistent for that special case too. 
Then long down the road of the development of a particular code some special 
case will use a segment of size zero, and it will be hard to trace back this 
error to the mpi library. 

Peter 


- Original Message -



Yes, that is what I meant. 

Enclosed is a C example. 
The point is that the code would logically make sense for task 0, but since it 
asks for a segment of size=0, it only gets a null pointer, which cannot be used 
to access the shared parts. 

Peter 

- Original Message -


I think Peter's point is that if 
- the windows uses contiguous memory 
*and* 
- all tasks knows how much memory was allocated by all other tasks in the 
window 
then it could/should be possible to get rid of MPI_Win_shared_query 

that is likely true if no task allocates zero byte. 
now, if a task allocates zero byte, MPI_Win_allocate_shared could return a null 
pointer and hence makes MPI_Win_shared_query usage mandatory. 

in his example, task 0 allocates zero bytes, so he was expecting the returned 
pointer on task zero points to the memory allocated by task 1. 

if "may enable" should be read as "does enable", then returning a null pointer 
can be seen as a bug. 
if "may enable" can be read as "does not always enable", the returning a null 
pointer is compliant with the standard. 

I am clearly not good at reading/interpreting the standard, so using 
MPI_Win_shared_query is my recommended way to get it work. 
(feel free to call it "bulletproof", "overkill", or even "right") 

Cheers, 

Gilles 

On Thursday, February 11, 2016, Jeff Hammond < jeff.scie...@gmail.com > wrote: 





On Wed, Feb 10, 2016 at 8:44 AM, Peter Wind < peter.w...@met.no > wrote: 



I agree that in practice the best practice would be to use Win_shared_query. 

Still I am confused by this part in the documentation: 
"The allocated memory is contiguous across process ranks unless the info key 
alloc_shared_noncontig is specified. Contiguous across process ranks means that 
the first address in the memory segment of process i is consecutive with the 
last address in the memory segment of process i - 1. This may enable the user 
to calculate remote address offsets with local information only." 

Isn't this an encouragement to use the pointer of Win_allocate_shared directly? 





No, it is not. Win_allocate_shared only gives you the pointer to the portion of 
the allocation that is owned by the calling process. If you want to access the 
whole slab, call Win_shared_query(..,rank=0,..) and use the resulting baseptr. 

I attempted to modify your code to be more correct, but I don't know enough 
Fortran to get it right. If you can parse C examples, I'll provide some of 
those. 

Jeff 




Peter 





I don't know about bulletproof, but Win_shared_query is the *only* valid way to 
get the addresses of memory in other processes associated with a window. 

The default for Win_allocate_shared is contiguous memory, but it can and likely 
will be mapped differently into each process, in which case only relative 
offsets are transferrable. 

Jeff 

On Wed, Feb 10, 2016 at 4:19 AM, Gilles Gouaillardet < 
gilles.gouaillar...@gmail.com > wrote: 


Peter, 

The bulletproof way is to use MPI_Win_shared_query after 
MPI_Win_allocate_shared. 
I do not know if current behavior is a bug or a feature... 

Cheers, 

Gilles 


On Wednesday, February 10, 2016, Peter Wind < peter.w...@met.no > wrote: 


Hi, 

Under fortran, MPI_Win_allocate_shared is called with a window size of zero for 
some processes. 
The output pointer is then not valid for these processes (null pointer). 
Did I understood this wrongly? shouldn't the pointers be contiguous, so that 
for a zero sized window, the pointer should point to the start of the segment 
of the next rank? 
The documentation explicitly specifies "size = 0 is valid". 

Attached a small code, where rank=0 allocate a window of size zero. All the 
other ranks get valid pointers, except rank 0. 

Best regards, 
Peter 
___ 
users mailing list 
us...@open-mpi.org 
Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users 
Link to this post: 
http://www.open-mpi.org/community/lists/users/2016/02/28485.php 




___ 
users mailing list 
us...@open-mpi.org 
Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users 
Link to this post:

Re: [OMPI users] shared memory zero size segment

2016-02-11 Thread Peter Wind 
Yes, that is what I meant. 

Enclosed is a C example. 
The point is that the code would logically make sense for task 0, but since it 
asks for a segment of size=0, it only gets a null pointer, which cannot be used 
to access the shared parts. 

Peter 

- Original Message -

> I think Peter's point is that if
> - the windows uses contiguous memory
> *and*
> - all tasks knows how much memory was allocated by all other tasks in the
> window
> then it could/should be possible to get rid of MPI_Win_shared_query

> that is likely true if no task allocates zero byte.
> now, if a task allocates zero byte, MPI_Win_allocate_shared could return a
> null pointer and hence makes MPI_Win_shared_query usage mandatory.

> in his example, task 0 allocates zero bytes, so he was expecting the returned
> pointer on task zero points to the memory allocated by task 1.

> if "may enable" should be read as "does enable", then returning a null
> pointer can be seen as a bug.
> if "may enable" can be read as "does not always enable", the returning a null
> pointer is compliant with the standard.

> I am clearly not good at reading/interpreting the standard, so using
> MPI_Win_shared_query is my recommended way to get it work.
> (feel free to call it "bulletproof", "overkill", or even "right")

> Cheers,

> Gilles

> On Thursday, February 11, 2016, Jeff Hammond < jeff.scie...@gmail.com >
> wrote:

> > On Wed, Feb 10, 2016 at 8:44 AM, Peter Wind < peter.w...@met.no > wrote:
> 

> > > I agree that in practice the best practice would be to use
> > > Win_shared_query.
> > 
> 

> > > Still I am confused by this part in the documentation:
> > 
> 
> > > "The allocated memory is contiguous across process ranks unless the info
> > > key
> > > alloc_shared_noncontig is specified. Contiguous across process ranks
> > > means
> > > that the first address in the memory segment of process i is consecutive
> > > with the last address in the memory segment of process i - 1. This may
> > > enable the user to calculate remote address offsets with local
> > > information
> > > only."
> > 
> 

> > > Isn't this an encouragement to use the pointer of Win_allocate_shared
> > > directly?
> > 
> 

> > No, it is not. Win_allocate_shared only gives you the pointer to the
> > portion
> > of the allocation that is owned by the calling process. If you want to
> > access the whole slab, call Win_shared_query(..,rank=0,..) and use the
> > resulting baseptr.
> 

> > I attempted to modify your code to be more correct, but I don't know enough
> > Fortran to get it right. If you can parse C examples, I'll provide some of
> > those.
> 

> > Jeff
> 

> > > Peter
> > 
> 

> > > > I don't know about bulletproof, but Win_shared_query is the *only*
> > > > valid
> > > > way
> > > > to get the addresses of memory in other processes associated with a
> > > > window.
> > > 
> > 
> 

> > > > The default for Win_allocate_shared is contiguous memory, but it can
> > > > and
> > > > likely will be mapped differently into each process, in which case only
> > > > relative offsets are transferrable.
> > > 
> > 
> 

> > > > Jeff
> > > 
> > 
> 

> > > > On Wed, Feb 10, 2016 at 4:19 AM, Gilles Gouaillardet <
> > > > gilles.gouaillar...@gmail.com > wrote:
> > > 
> > 
> 

> > > > > Peter,
> > > > 
> > > 
> > 
> 

> > > > > The bulletproof way is to use MPI_Win_shared_query after
> > > > > MPI_Win_allocate_shared.
> > > > 
> > > 
> > 
> 
> > > > > I do not know if current behavior is a bug or a feature...
> > > > 
> > > 
> > 
> 

> > > > > Cheers,
> > > > 
> > > 
> > 
> 

> > > > > Gilles
> > > > 
> > > 
> > 
> 

> > > > > On Wednesday, February 10, 2016, Peter Wind < peter.w...@met.no >
> > > > > wrote:
> > > > 
> > > 
> > 
> 

> > > > > > Hi,
> > > > > 
> > > > 
> > > 
> > 
> 

> > > > > > Under fortran, MPI_Win_allocate_shared is called with a window size
> > > > > > of
> > > > > > zero
> > > > > > for some processes.
> > > > > 
> > > > 
> > > 
> > 
> 
> > > > > > The output pointer is then not valid for these processes (null
> > > > > > pointer).
> > > > > 
> > > > 
> > > 
> > 
> 
> > > > > > Did I understood this wrongly? shouldn't the pointers be
> > > > > > contiguous,
> > > > > > so
> > > > > > that
> > > > > > for a zero sized window, the pointer should point to the start of
> > > > > > the
> > > > > > segment of the next rank?
> > > > > 
> > > > 
> > > 
> > 
> 
> > > > > > The documentation explicitly specifies "size = 0 is valid".
> > > > > 
> > > > 
> > > 
> > 
> 

> > > > > > Attached a small code, where rank=0 allocate a window of size zero.
> > > > > > All
> > > > > > the
> > > > > > other ranks get valid pointers, except rank 0.
> > > > > 
> > > > 
> > > 
> > 
> 

> > > > > > Best regards,
> > > > > 
> > > > 
> > > 
> > 
> 
> > > > > > Peter
> > > > > 
> > > > 
> > > 
> > 
> 
> > > > > > ___
> > > > > 
> > > > 
> > > 
> > 
> 
> > > > > > users mailing list
> > > > > 
> > > > 
> > > 
> > 
> 
> > > > > > us...@open-mpi.org
>

Re: [OMPI users] shared memory zero size segment

2016-02-10 Thread Gilles Gouaillardet 
I think Peter's point is that if
- the windows uses contiguous memory
*and*
- all tasks knows how much memory was allocated by all other tasks in the
window
then it could/should be possible to get rid of MPI_Win_shared_query

that is likely true if no task allocates zero byte.
now, if a task allocates zero byte, MPI_Win_allocate_shared could return a
null pointer and hence makes MPI_Win_shared_query usage mandatory.

in his example, task 0 allocates zero bytes, so he was expecting the
returned pointer on task zero points to the memory allocated by task 1.

if "may enable" should be read as "does enable", then returning a null
pointer can be seen as a bug.
if "may enable" can be read as "does not always enable", the returning a
null pointer is compliant with the standard.

I am clearly not good at reading/interpreting the standard, so using
MPI_Win_shared_query is my recommended way to get it work.
(feel free to call it "bulletproof",  "overkill", or even "right")

Cheers,

Gilles

On Thursday, February 11, 2016, Jeff Hammond  wrote:

>
>
> On Wed, Feb 10, 2016 at 8:44 AM, Peter Wind  > wrote:
>
>> I agree that in practice the best practice would be to use
>> Win_shared_query.
>>
>> Still I am confused by this part in the documentation:
>> "The allocated memory is contiguous across process ranks unless the info
>> key *alloc_shared_noncontig* is specified. Contiguous across process
>> ranks means that the first address in the memory segment of process i is
>> consecutive with the last address in the memory segment of process i - 1.
>> This may enable the user to calculate remote address offsets with local
>> information only."
>>
>> Isn't this an encouragement to use the pointer of Win_allocate_shared
>> directly?
>>
>>
> No, it is not.  Win_allocate_shared only gives you the pointer to the
> portion of the allocation that is owned by the calling process.  If you
> want to access the whole slab, call Win_shared_query(..,rank=0,..) and use
> the resulting baseptr.
>
> I attempted to modify your code to be more correct, but I don't know
> enough Fortran to get it right.  If you can parse C examples, I'll provide
> some of those.
>
> Jeff
>
>
>> Peter
>>
>> --
>>
>> I don't know about bulletproof, but Win_shared_query is the *only* valid
>> way to get the addresses of memory in other processes associated with a
>> window.
>>
>> The default for Win_allocate_shared is contiguous memory, but it can and
>> likely will be mapped differently into each process, in which case only
>> relative offsets are transferrable.
>>
>> Jeff
>>
>> On Wed, Feb 10, 2016 at 4:19 AM, Gilles Gouaillardet <
>> gilles.gouaillar...@gmail.com
>> > wrote:
>>
>>> Peter,
>>>
>>> The bulletproof way is to use MPI_Win_shared_query after
>>> MPI_Win_allocate_shared.
>>> I do not know if current behavior is a bug or a feature...
>>>
>>> Cheers,
>>>
>>> Gilles
>>>
>>>
>>> On Wednesday, February 10, 2016, Peter Wind >> > wrote:
>>>
 Hi,

 Under fortran, MPI_Win_allocate_shared is called with a window size of
 zero for some processes.
 The output pointer is then not valid for these processes (null pointer).
 Did I understood this wrongly? shouldn't the pointers be contiguous, so
 that for a zero sized window, the pointer should point to the start of the
 segment of the next rank?
 The documentation explicitly specifies "size = 0 is valid".

 Attached a small code, where rank=0 allocate a window of size zero. All
 the other ranks get valid pointers, except rank 0.

 Best regards,
 Peter
 ___
 users mailing list
 us...@open-mpi.org
 Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
 Link to this post:
 http://www.open-mpi.org/community/lists/users/2016/02/28485.php

>>>
>>> ___
>>> users mailing list
>>> us...@open-mpi.org 
>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
>>> Link to this post:
>>> http://www.open-mpi.org/community/lists/users/2016/02/28493.php
>>>
>>
>>
>>
>> --
>> Jeff Hammond
>> jeff.scie...@gmail.com
>> 
>> http://jeffhammond.github.io/
>>
>> ___
>> users mailing list
>> us...@open-mpi.org 
>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
>> Link to this post:
>> http://www.open-mpi.org/community/lists/users/2016/02/28496.php
>>
>>
>>
>> ___
>> users mailing list
>> us...@open-mpi.org

Re: [OMPI users] shared memory zero size segment

2016-02-10 Thread Jeff Hammond 
On Wed, Feb 10, 2016 at 8:44 AM, Peter Wind  wrote:

> I agree that in practice the best practice would be to use
> Win_shared_query.
>
> Still I am confused by this part in the documentation:
> "The allocated memory is contiguous across process ranks unless the info
> key *alloc_shared_noncontig* is specified. Contiguous across process
> ranks means that the first address in the memory segment of process i is
> consecutive with the last address in the memory segment of process i - 1.
> This may enable the user to calculate remote address offsets with local
> information only."
>
> Isn't this an encouragement to use the pointer of Win_allocate_shared
> directly?
>
>
No, it is not.  Win_allocate_shared only gives you the pointer to the
portion of the allocation that is owned by the calling process.  If you
want to access the whole slab, call Win_shared_query(..,rank=0,..) and use
the resulting baseptr.

I attempted to modify your code to be more correct, but I don't know enough
Fortran to get it right.  If you can parse C examples, I'll provide some of
those.

Jeff


> Peter
>
> --
>
> I don't know about bulletproof, but Win_shared_query is the *only* valid
> way to get the addresses of memory in other processes associated with a
> window.
>
> The default for Win_allocate_shared is contiguous memory, but it can and
> likely will be mapped differently into each process, in which case only
> relative offsets are transferrable.
>
> Jeff
>
> On Wed, Feb 10, 2016 at 4:19 AM, Gilles Gouaillardet <
> gilles.gouaillar...@gmail.com> wrote:
>
>> Peter,
>>
>> The bulletproof way is to use MPI_Win_shared_query after
>> MPI_Win_allocate_shared.
>> I do not know if current behavior is a bug or a feature...
>>
>> Cheers,
>>
>> Gilles
>>
>>
>> On Wednesday, February 10, 2016, Peter Wind  wrote:
>>
>>> Hi,
>>>
>>> Under fortran, MPI_Win_allocate_shared is called with a window size of
>>> zero for some processes.
>>> The output pointer is then not valid for these processes (null pointer).
>>> Did I understood this wrongly? shouldn't the pointers be contiguous, so
>>> that for a zero sized window, the pointer should point to the start of the
>>> segment of the next rank?
>>> The documentation explicitly specifies "size = 0 is valid".
>>>
>>> Attached a small code, where rank=0 allocate a window of size zero. All
>>> the other ranks get valid pointers, except rank 0.
>>>
>>> Best regards,
>>> Peter
>>> ___
>>> users mailing list
>>> us...@open-mpi.org
>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
>>> Link to this post:
>>> http://www.open-mpi.org/community/lists/users/2016/02/28485.php
>>>
>>
>> ___
>> users mailing list
>> us...@open-mpi.org
>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
>> Link to this post:
>> http://www.open-mpi.org/community/lists/users/2016/02/28493.php
>>
>
>
>
> --
> Jeff Hammond
> jeff.scie...@gmail.com
> http://jeffhammond.github.io/
>
> ___
> users mailing list
> us...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> Link to this post:
> http://www.open-mpi.org/community/lists/users/2016/02/28496.php
>
>
>
> ___
> users mailing list
> us...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> Link to this post:
> http://www.open-mpi.org/community/lists/users/2016/02/28497.php
>



-- 
Jeff Hammond
jeff.scie...@gmail.com
http://jeffhammond.github.io/


sharetest.f90
Description: Binary data

Re: [OMPI users] shared memory zero size segment

2016-02-10 Thread Peter Wind 
I agree that in practice the best practice would be to use Win_shared_query. 

Still I am confused by this part in the documentation: 
"The allocated memory is contiguous across process ranks unless the info key 
alloc_shared_noncontig is specified. Contiguous across process ranks means that 
the first address in the memory segment of process i is consecutive with the 
last address in the memory segment of process i - 1. This may enable the user 
to calculate remote address offsets with local information only." 

Isn't this an encouragement to use the pointer of Win_allocate_shared directly? 

Peter 

- Original Message -

> I don't know about bulletproof, but Win_shared_query is the *only* valid way
> to get the addresses of memory in other processes associated with a window.

> The default for Win_allocate_shared is contiguous memory, but it can and
> likely will be mapped differently into each process, in which case only
> relative offsets are transferrable.

> Jeff

> On Wed, Feb 10, 2016 at 4:19 AM, Gilles Gouaillardet <
> gilles.gouaillar...@gmail.com > wrote:

> > Peter,
> 

> > The bulletproof way is to use MPI_Win_shared_query after
> > MPI_Win_allocate_shared.
> 
> > I do not know if current behavior is a bug or a feature...
> 

> > Cheers,
> 

> > Gilles
> 

> > On Wednesday, February 10, 2016, Peter Wind < peter.w...@met.no > wrote:
> 

> > > Hi,
> > 
> 

> > > Under fortran, MPI_Win_allocate_shared is called with a window size of
> > > zero
> > > for some processes.
> > 
> 
> > > The output pointer is then not valid for these processes (null pointer).
> > 
> 
> > > Did I understood this wrongly? shouldn't the pointers be contiguous, so
> > > that
> > > for a zero sized window, the pointer should point to the start of the
> > > segment of the next rank?
> > 
> 
> > > The documentation explicitly specifies "size = 0 is valid".
> > 
> 

> > > Attached a small code, where rank=0 allocate a window of size zero. All
> > > the
> > > other ranks get valid pointers, except rank 0.
> > 
> 

> > > Best regards,
> > 
> 
> > > Peter
> > 
> 
> > > ___
> > 
> 
> > > users mailing list
> > 
> 
> > > us...@open-mpi.org
> > 
> 
> > > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> > 
> 
> > > Link to this post:
> > > http://www.open-mpi.org/community/lists/users/2016/02/28485.php
> > 
> 

> > ___
> 
> > users mailing list
> 
> > us...@open-mpi.org
> 
> > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> 
> > Link to this post:
> > http://www.open-mpi.org/community/lists/users/2016/02/28493.php
> 

> --
> Jeff Hammond
> jeff.scie...@gmail.com
> http://jeffhammond.github.io/

> ___
> users mailing list
> us...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> Link to this post:
> http://www.open-mpi.org/community/lists/users/2016/02/28496.php

Re: [OMPI users] shared memory zero size segment

2016-02-10 Thread Jeff Hammond 
I don't know about bulletproof, but Win_shared_query is the *only* valid
way to get the addresses of memory in other processes associated with a
window.

The default for Win_allocate_shared is contiguous memory, but it can and
likely will be mapped differently into each process, in which case only
relative offsets are transferrable.

Jeff

On Wed, Feb 10, 2016 at 4:19 AM, Gilles Gouaillardet <
gilles.gouaillar...@gmail.com> wrote:

> Peter,
>
> The bulletproof way is to use MPI_Win_shared_query after
> MPI_Win_allocate_shared.
> I do not know if current behavior is a bug or a feature...
>
> Cheers,
>
> Gilles
>
>
> On Wednesday, February 10, 2016, Peter Wind  wrote:
>
>> Hi,
>>
>> Under fortran, MPI_Win_allocate_shared is called with a window size of
>> zero for some processes.
>> The output pointer is then not valid for these processes (null pointer).
>> Did I understood this wrongly? shouldn't the pointers be contiguous, so
>> that for a zero sized window, the pointer should point to the start of the
>> segment of the next rank?
>> The documentation explicitly specifies "size = 0 is valid".
>>
>> Attached a small code, where rank=0 allocate a window of size zero. All
>> the other ranks get valid pointers, except rank 0.
>>
>> Best regards,
>> Peter
>> ___
>> users mailing list
>> us...@open-mpi.org
>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
>> Link to this post:
>> http://www.open-mpi.org/community/lists/users/2016/02/28485.php
>>
>
> ___
> users mailing list
> us...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> Link to this post:
> http://www.open-mpi.org/community/lists/users/2016/02/28493.php
>



-- 
Jeff Hammond
jeff.scie...@gmail.com
http://jeffhammond.github.io/

Re: [OMPI users] shared memory zero size segment

2016-02-10 Thread Gilles Gouaillardet 
Peter,

The bulletproof way is to use MPI_Win_shared_query after
MPI_Win_allocate_shared.
I do not know if current behavior is a bug or a feature...

Cheers,

Gilles

On Wednesday, February 10, 2016, Peter Wind  wrote:

> Hi,
>
> Under fortran, MPI_Win_allocate_shared is called with a window size of
> zero for some processes.
> The output pointer is then not valid for these processes (null pointer).
> Did I understood this wrongly? shouldn't the pointers be contiguous, so
> that for a zero sized window, the pointer should point to the start of the
> segment of the next rank?
> The documentation explicitly specifies "size = 0 is valid".
>
> Attached a small code, where rank=0 allocate a window of size zero. All
> the other ranks get valid pointers, except rank 0.
>
> Best regards,
> Peter
> ___
> users mailing list
> us...@open-mpi.org 
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> Link to this post:
> http://www.open-mpi.org/community/lists/users/2016/02/28485.php
>

Re: [OMPI users] shared memory zero size segment

2016-02-10 Thread Peter Wind 
Sorry for that, here is the attachement!

Peter

- Original Message -
> Peter --
> 
> Somewhere along the way, your attachment got lost.  Could you re-send?
> 
> Thanks.
> 
> 
> > On Feb 10, 2016, at 5:56 AM, Peter Wind  wrote:
> > 
> > Hi,
> > 
> > Under fortran, MPI_Win_allocate_shared is called with a window size of zero
> > for some processes.
> > The output pointer is then not valid for these processes (null pointer).
> > Did I understood this wrongly? shouldn't the pointers be contiguous, so
> > that for a zero sized window, the pointer should point to the start of the
> > segment of the next rank?
> > The documentation explicitly specifies "size = 0 is valid".
> > 
> > Attached a small code, where rank=0 allocate a window of size zero. All the
> > other ranks get valid pointers, except rank 0.
> > 
> > Best regards,
> > Peter
> > ___
> > users mailing list
> > us...@open-mpi.org
> > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> > Link to this post:
> > http://www.open-mpi.org/community/lists/users/2016/02/28485.php
> 
> 
> --
> Jeff Squyres
> jsquy...@cisco.com
> For corporate legal information go to:
> http://www.cisco.com/web/about/doing_business/legal/cri/
> 
> ___
> users mailing list
> us...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> Link to this post:
> http://www.open-mpi.org/community/lists/users/2016/02/28486.php
> 
program sharetest

! test zero size segment.
! run on at least 3 cpus
! mpirun -np 4 a.out

   use mpi

   use, intrinsic :: iso_c_binding,

   implicit none


   integer, parameter :: nsize = 20
   integer, pointer   :: array(:)
   integer:: num_procs
   integer:: ierr
   integer:: irank, irank_group
   integer:: win
   integer:: disp_unit
   type(c_ptr):: cp1
   type(c_ptr):: cp2

   integer(MPI_ADDRESS_KIND) :: win_size
   integer(MPI_ADDRESS_KIND) :: segment_size

   call MPI_Init(ierr)
   call MPI_Comm_size(MPI_COMM_WORLD, num_procs, ierr)
   call MPI_Comm_rank(MPI_COMM_WORLD, irank, ierr)

   disp_unit = sizeof(1)

   win_size = irank*disp_unit

   call MPI_Win_allocate_shared(win_size, disp_unit, MPI_INFO_NULL, MPI_COMM_WORLD, cp1, win, ierr)

!   write(*,*)'rank ', irank,', pointer ',cp1

  call c_f_pointer(cp1, array, [nsize])

77 format(4(A,I3))

   if(irank/=0)then
  array(1)=irank
  CALL MPI_BARRIER(MPI_COMM_WORLD, ierr)
  if(irank/=num_procs-1)then
 print 77, ' rank', irank, ':  array(1)', array(1),' shared with next rank: ',array(irank+1)
  else
 print 77, ' rank', irank, ':  array(1)', array(1),' shared with previous rank: ',array(0)
  endif
  CALL MPI_BARRIER(MPI_COMM_WORLD, ierr)
   else
 CALL MPI_BARRIER(MPI_COMM_WORLD, ierr)
  CALL MPI_BARRIER(MPI_COMM_WORLD, ierr)
 if(.not.associated(array))then
print 77, 'zero pointer found, rank', irank
 else
print 77, ' rank', irank, ' array associated '
print 77, ' rank', irank, ':  array(1) ', array(1),' shared with next rank: ',array(irank+1)
 endif
   endif


   call MPI_Finalize(ierr)

 end program sharetest

Re: [OMPI users] shared memory zero size segment

2016-02-10 Thread Peter Wind 


- Original Message -
> Peter --
> 
> Somewhere along the way, your attachment got lost.  Could you re-send?
> 
> Thanks.
> 
> 
> > On Feb 10, 2016, at 5:56 AM, Peter Wind  wrote:
> > 
> > Hi,
> > 
> > Under fortran, MPI_Win_allocate_shared is called with a window size of zero
> > for some processes.
> > The output pointer is then not valid for these processes (null pointer).
> > Did I understood this wrongly? shouldn't the pointers be contiguous, so
> > that for a zero sized window, the pointer should point to the start of the
> > segment of the next rank?
> > The documentation explicitly specifies "size = 0 is valid".
> > 
> > Attached a small code, where rank=0 allocate a window of size zero. All the
> > other ranks get valid pointers, except rank 0.
> > 
> > Best regards,
> > Peter
> > ___
> > users mailing list
> > us...@open-mpi.org
> > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> > Link to this post:
> > http://www.open-mpi.org/community/lists/users/2016/02/28485.php
> 
> 
> --
> Jeff Squyres
> jsquy...@cisco.com
> For corporate legal information go to:
> http://www.cisco.com/web/about/doing_business/legal/cri/
> 
> ___
> users mailing list
> us...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> Link to this post:
> http://www.open-mpi.org/community/lists/users/2016/02/28486.php
> 
program sharetest

! test zero size segment.
! run on at least 3 cpus
! mpirun -np 4 a.out

   use mpi

   use, intrinsic :: iso_c_binding,

   implicit none


   integer, parameter :: nsize = 20
   integer, pointer   :: array(:)
   integer:: num_procs
   integer:: ierr
   integer:: irank, irank_group
   integer:: win
   integer:: disp_unit
   type(c_ptr):: cp1
   type(c_ptr):: cp2

   integer(MPI_ADDRESS_KIND) :: win_size
   integer(MPI_ADDRESS_KIND) :: segment_size

   call MPI_Init(ierr)
   call MPI_Comm_size(MPI_COMM_WORLD, num_procs, ierr)
   call MPI_Comm_rank(MPI_COMM_WORLD, irank, ierr)

   disp_unit = sizeof(1)

   win_size = irank*disp_unit

   call MPI_Win_allocate_shared(win_size, disp_unit, MPI_INFO_NULL, MPI_COMM_WORLD, cp1, win, ierr)

!   write(*,*)'rank ', irank,', pointer ',cp1

  call c_f_pointer(cp1, array, [nsize])

77 format(4(A,I3))

   if(irank/=0)then
  array(1)=irank
  CALL MPI_BARRIER(MPI_COMM_WORLD, ierr)
  if(irank/=num_procs-1)then
 print 77, ' rank', irank, ':  array(1)', array(1),' shared with next rank: ',array(irank+1)
  else
 print 77, ' rank', irank, ':  array(1)', array(1),' shared with previous rank: ',array(0)
  endif
  CALL MPI_BARRIER(MPI_COMM_WORLD, ierr)
   else
 CALL MPI_BARRIER(MPI_COMM_WORLD, ierr)
  CALL MPI_BARRIER(MPI_COMM_WORLD, ierr)
 if(.not.associated(array))then
print 77, 'zero pointer found, rank', irank
 else
print 77, ' rank', irank, ' array associated '
print 77, ' rank', irank, ':  array(1) ', array(1),' shared with next rank: ',array(irank+1)
 endif
   endif


   call MPI_Finalize(ierr)

 end program sharetest

Re: [OMPI users] shared memory zero size segment

2016-02-10 Thread Jeff Squyres (jsquyres) 
Peter --

Somewhere along the way, your attachment got lost.  Could you re-send?

Thanks.


> On Feb 10, 2016, at 5:56 AM, Peter Wind  wrote:
> 
> Hi,
> 
> Under fortran, MPI_Win_allocate_shared is called with a window size of zero 
> for some processes.
> The output pointer is then not valid for these processes (null pointer).
> Did I understood this wrongly? shouldn't the pointers be contiguous, so that 
> for a zero sized window, the pointer should point to the start of the segment 
> of the next rank?
> The documentation explicitly specifies "size = 0 is valid".
> 
> Attached a small code, where rank=0 allocate a window of size zero. All the 
> other ranks get valid pointers, except rank 0.
> 
> Best regards,
> Peter
> ___
> users mailing list
> us...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> Link to this post: 
> http://www.open-mpi.org/community/lists/users/2016/02/28485.php


-- 
Jeff Squyres
jsquy...@cisco.com
For corporate legal information go to: 
http://www.cisco.com/web/about/doing_business/legal/cri/