Re: [OMPI users] shared memory zero size segment
No clarification necessary. Standard is not user guide. Semantics are clear from what is defined. Users who don't like the interface can write a library that does what they want. Jeff On Thursday, February 11, 2016, Nathan Hjelm wrote: > > I should also say that I think this is something that may be worth > clarifying in the standard. Either semantic is fine with me but there is > no reason to change the behavior if it does not violate the standard. > > -Nathan > > On Thu, Feb 11, 2016 at 01:35:28PM -0700, Nathan Hjelm wrote: > > > > Jeff probably ran with MPICH. Open MPI's are consistent with our choice > > of definition for size=0: > > > > query: me=1, them=0, size=0, disp=1, base=0x0 > > query: me=1, them=1, size=4, disp=1, base=0x1097e30f8 > > query: me=1, them=2, size=4, disp=1, base=0x1097e30fc > > query: me=1, them=3, size=4, disp=1, base=0x1097e3100 > > query: me=2, them=0, size=0, disp=1, base=0x0 > > query: me=2, them=1, size=4, disp=1, base=0x109fe10f8 > > query: me=2, them=2, size=4, disp=1, base=0x109fe10fc > > query: me=2, them=3, size=4, disp=1, base=0x109fe1100 > > query: me=2, them=PROC_NULL, size=4, disp=1, base=0x109fe10f8 > > query: me=3, them=0, size=0, disp=1, base=0x0 > > query: me=3, them=1, size=4, disp=1, base=0x1088f30f8 > > query: me=3, them=2, size=4, disp=1, base=0x1088f30fc > > query: me=3, them=3, size=4, disp=1, base=0x1088f3100 > > query: me=3, them=PROC_NULL, size=4, disp=1, base=0x1088f30f8 > > query: me=0, them=0, size=0, disp=1, base=0x0 > > query: me=0, them=1, size=4, disp=1, base=0x105b890f8 > > query: me=0, them=2, size=4, disp=1, base=0x105b890fc > > query: me=0, them=3, size=4, disp=1, base=0x105b89100 > > query: me=0, them=PROC_NULL, size=4, disp=1, base=0x105b890f8 > > query: me=1, them=PROC_NULL, size=4, disp=1, base=0x1097e30f8 > > > > To be portable only use MPI_Win_shared_query and do not rely on the > > return value of base if you pass size = 0. > > > > -Nathan > > > > On Thu, Feb 11, 2016 at 08:23:16PM +, Peter Wind wrote: > > >Thanks Jeff, that was an interesting result. The pointers are here > well > > >defined, also for the zero size segment. > > >However I can't reproduce your output. I still get null pointers > (output > > >below). > > >(I tried both 1.8.5 and 1.10.2 versions) > > >What could be the difference? > > >Peter > > >mpirun -np 4 a.out > > >query: me=0, them=0, size=0, disp=1, base=(nil) > > >query: me=0, them=1, size=4, disp=1, base=0x2aee280030d0 > > >query: me=0, them=2, size=4, disp=1, base=0x2aee280030d4 > > >query: me=0, them=3, size=4, disp=1, base=0x2aee280030d8 > > >query: me=0, them=PROC_NULL, size=4, disp=1, base=0x2aee280030d0 > > >query: me=1, them=0, size=0, disp=1, base=(nil) > > >query: me=1, them=1, size=4, disp=1, base=0x2aabbb9ce0d0 > > >query: me=1, them=2, size=4, disp=1, base=0x2aabbb9ce0d4 > > >query: me=1, them=3, size=4, disp=1, base=0x2aabbb9ce0d8 > > >query: me=1, them=PROC_NULL, size=4, disp=1, base=0x2aabbb9ce0d0 > > >query: me=2, them=0, size=0, disp=1, base=(nil) > > >query: me=2, them=1, size=4, disp=1, base=0x2b1579dd40d0 > > >query: me=2, them=2, size=4, disp=1, base=0x2b1579dd40d4 > > >query: me=2, them=3, size=4, disp=1, base=0x2b1579dd40d8 > > >query: me=2, them=PROC_NULL, size=4, disp=1, base=0x2b1579dd40d0 > > >query: me=3, them=0, size=0, disp=1, base=(nil) > > >query: me=3, them=1, size=4, disp=1, base=0x2ac8d2c350d0 > > >query: me=3, them=2, size=4, disp=1, base=0x2ac8d2c350d4 > > >query: me=3, them=3, size=4, disp=1, base=0x2ac8d2c350d8 > > >query: me=3, them=PROC_NULL, size=4, disp=1, base=0x2ac8d2c350d0 > > > > > > > -- > > > > > > See attached. Output below. Note that the base you get for > ranks 0 and > > > 1 is the same, so you need to use the fact that size=0 at rank=0 > to know > > > not to dereference that pointer and expect to be writing into > rank 0's > > > memory, since you will write into rank 1's. > > > I would probably add "if (size==0) base=NULL;" for good measure. > > > Jeff > > > > > > $ mpirun -n 4 ./a.out > > > > > > query: me=0, them=0, size=0, disp=1, base=0x10bd64000 > > > > > > query: me=0, them=1, size=4, disp=1, base=0x10bd64000 > > > > > > query: me=0, them=2, size=4, disp=1, base=0x10bd64004 > > > > > > query: me=0, them=3, size=4, disp=1, base=0x10bd64008 > > > > > > query: me=0, them=PROC_NULL, size=4, disp=1, base=0x10bd64000 > > > > > > query: me=1, them=0, size=0, disp=1, base=0x102d3b000 > > > > > > query: me=1, them=1, size=4, disp=1, base=0x102d3b000 > > > > > > query: me=1, them=2, size=4, disp=1, base=0x102d3b004 > > > > > > query: me=1, them=3, size=4, disp=1, base=0x102d3b008 > > > > > > query: me=1, them=PROC_NULL, size=4, disp=1, base=0x102d3b000 > > > > > > query: me=2, them=0, siz
Re: [OMPI users] shared memory zero size segment
Indeed, I ran with MPICH. But I like OpenMPI's choice better here, which is why I said that I would explicitly set the pointer to bull when size is zero. Jeff On Thursday, February 11, 2016, Nathan Hjelm wrote: > > Jeff probably ran with MPICH. Open MPI's are consistent with our choice > of definition for size=0: > > query: me=1, them=0, size=0, disp=1, base=0x0 > query: me=1, them=1, size=4, disp=1, base=0x1097e30f8 > query: me=1, them=2, size=4, disp=1, base=0x1097e30fc > query: me=1, them=3, size=4, disp=1, base=0x1097e3100 > query: me=2, them=0, size=0, disp=1, base=0x0 > query: me=2, them=1, size=4, disp=1, base=0x109fe10f8 > query: me=2, them=2, size=4, disp=1, base=0x109fe10fc > query: me=2, them=3, size=4, disp=1, base=0x109fe1100 > query: me=2, them=PROC_NULL, size=4, disp=1, base=0x109fe10f8 > query: me=3, them=0, size=0, disp=1, base=0x0 > query: me=3, them=1, size=4, disp=1, base=0x1088f30f8 > query: me=3, them=2, size=4, disp=1, base=0x1088f30fc > query: me=3, them=3, size=4, disp=1, base=0x1088f3100 > query: me=3, them=PROC_NULL, size=4, disp=1, base=0x1088f30f8 > query: me=0, them=0, size=0, disp=1, base=0x0 > query: me=0, them=1, size=4, disp=1, base=0x105b890f8 > query: me=0, them=2, size=4, disp=1, base=0x105b890fc > query: me=0, them=3, size=4, disp=1, base=0x105b89100 > query: me=0, them=PROC_NULL, size=4, disp=1, base=0x105b890f8 > query: me=1, them=PROC_NULL, size=4, disp=1, base=0x1097e30f8 > > To be portable only use MPI_Win_shared_query and do not rely on the > return value of base if you pass size = 0. > > -Nathan > > On Thu, Feb 11, 2016 at 08:23:16PM +, Peter Wind wrote: > >Thanks Jeff, that was an interesting result. The pointers are here > well > >defined, also for the zero size segment. > >However I can't reproduce your output. I still get null pointers > (output > >below). > >(I tried both 1.8.5 and 1.10.2 versions) > >What could be the difference? > >Peter > >mpirun -np 4 a.out > >query: me=0, them=0, size=0, disp=1, base=(nil) > >query: me=0, them=1, size=4, disp=1, base=0x2aee280030d0 > >query: me=0, them=2, size=4, disp=1, base=0x2aee280030d4 > >query: me=0, them=3, size=4, disp=1, base=0x2aee280030d8 > >query: me=0, them=PROC_NULL, size=4, disp=1, base=0x2aee280030d0 > >query: me=1, them=0, size=0, disp=1, base=(nil) > >query: me=1, them=1, size=4, disp=1, base=0x2aabbb9ce0d0 > >query: me=1, them=2, size=4, disp=1, base=0x2aabbb9ce0d4 > >query: me=1, them=3, size=4, disp=1, base=0x2aabbb9ce0d8 > >query: me=1, them=PROC_NULL, size=4, disp=1, base=0x2aabbb9ce0d0 > >query: me=2, them=0, size=0, disp=1, base=(nil) > >query: me=2, them=1, size=4, disp=1, base=0x2b1579dd40d0 > >query: me=2, them=2, size=4, disp=1, base=0x2b1579dd40d4 > >query: me=2, them=3, size=4, disp=1, base=0x2b1579dd40d8 > >query: me=2, them=PROC_NULL, size=4, disp=1, base=0x2b1579dd40d0 > >query: me=3, them=0, size=0, disp=1, base=(nil) > >query: me=3, them=1, size=4, disp=1, base=0x2ac8d2c350d0 > >query: me=3, them=2, size=4, disp=1, base=0x2ac8d2c350d4 > >query: me=3, them=3, size=4, disp=1, base=0x2ac8d2c350d8 > >query: me=3, them=PROC_NULL, size=4, disp=1, base=0x2ac8d2c350d0 > > > > > -- > > > > See attached. Output below. Note that the base you get for ranks > 0 and > > 1 is the same, so you need to use the fact that size=0 at rank=0 to > know > > not to dereference that pointer and expect to be writing into rank > 0's > > memory, since you will write into rank 1's. > > I would probably add "if (size==0) base=NULL;" for good measure. > > Jeff > > > > $ mpirun -n 4 ./a.out > > > > query: me=0, them=0, size=0, disp=1, base=0x10bd64000 > > > > query: me=0, them=1, size=4, disp=1, base=0x10bd64000 > > > > query: me=0, them=2, size=4, disp=1, base=0x10bd64004 > > > > query: me=0, them=3, size=4, disp=1, base=0x10bd64008 > > > > query: me=0, them=PROC_NULL, size=4, disp=1, base=0x10bd64000 > > > > query: me=1, them=0, size=0, disp=1, base=0x102d3b000 > > > > query: me=1, them=1, size=4, disp=1, base=0x102d3b000 > > > > query: me=1, them=2, size=4, disp=1, base=0x102d3b004 > > > > query: me=1, them=3, size=4, disp=1, base=0x102d3b008 > > > > query: me=1, them=PROC_NULL, size=4, disp=1, base=0x102d3b000 > > > > query: me=2, them=0, size=0, disp=1, base=0x10aac1000 > > > > query: me=2, them=1, size=4, disp=1, base=0x10aac1000 > > > > query: me=2, them=2, size=4, disp=1, base=0x10aac1004 > > > > query: me=2, them=3, size=4, disp=1, base=0x10aac1008 > > > > query: me=2, them=PROC_NULL, size=4, disp=1, base=0x10aac1000 > > > > query: me=3, them=0, size=0, disp=1, base=0x100fa2000 > > > > query: me=3, them=1, size=4, disp=1, base=0x100fa2000 > > > > query: me=3, them=2, size=4, disp=1, base=0x100fa200
Re: [OMPI users] shared memory zero size segment
I should also say that I think this is something that may be worth clarifying in the standard. Either semantic is fine with me but there is no reason to change the behavior if it does not violate the standard. -Nathan On Thu, Feb 11, 2016 at 01:35:28PM -0700, Nathan Hjelm wrote: > > Jeff probably ran with MPICH. Open MPI's are consistent with our choice > of definition for size=0: > > query: me=1, them=0, size=0, disp=1, base=0x0 > query: me=1, them=1, size=4, disp=1, base=0x1097e30f8 > query: me=1, them=2, size=4, disp=1, base=0x1097e30fc > query: me=1, them=3, size=4, disp=1, base=0x1097e3100 > query: me=2, them=0, size=0, disp=1, base=0x0 > query: me=2, them=1, size=4, disp=1, base=0x109fe10f8 > query: me=2, them=2, size=4, disp=1, base=0x109fe10fc > query: me=2, them=3, size=4, disp=1, base=0x109fe1100 > query: me=2, them=PROC_NULL, size=4, disp=1, base=0x109fe10f8 > query: me=3, them=0, size=0, disp=1, base=0x0 > query: me=3, them=1, size=4, disp=1, base=0x1088f30f8 > query: me=3, them=2, size=4, disp=1, base=0x1088f30fc > query: me=3, them=3, size=4, disp=1, base=0x1088f3100 > query: me=3, them=PROC_NULL, size=4, disp=1, base=0x1088f30f8 > query: me=0, them=0, size=0, disp=1, base=0x0 > query: me=0, them=1, size=4, disp=1, base=0x105b890f8 > query: me=0, them=2, size=4, disp=1, base=0x105b890fc > query: me=0, them=3, size=4, disp=1, base=0x105b89100 > query: me=0, them=PROC_NULL, size=4, disp=1, base=0x105b890f8 > query: me=1, them=PROC_NULL, size=4, disp=1, base=0x1097e30f8 > > To be portable only use MPI_Win_shared_query and do not rely on the > return value of base if you pass size = 0. > > -Nathan > > On Thu, Feb 11, 2016 at 08:23:16PM +, Peter Wind wrote: > >Thanks Jeff, that was an interesting result. The pointers are here well > >defined, also for the zero size segment. > >However I can't reproduce your output. I still get null pointers (output > >below). > >(I tried both 1.8.5 and 1.10.2 versions) > >What could be the difference? > >Peter > >mpirun -np 4 a.out > >query: me=0, them=0, size=0, disp=1, base=(nil) > >query: me=0, them=1, size=4, disp=1, base=0x2aee280030d0 > >query: me=0, them=2, size=4, disp=1, base=0x2aee280030d4 > >query: me=0, them=3, size=4, disp=1, base=0x2aee280030d8 > >query: me=0, them=PROC_NULL, size=4, disp=1, base=0x2aee280030d0 > >query: me=1, them=0, size=0, disp=1, base=(nil) > >query: me=1, them=1, size=4, disp=1, base=0x2aabbb9ce0d0 > >query: me=1, them=2, size=4, disp=1, base=0x2aabbb9ce0d4 > >query: me=1, them=3, size=4, disp=1, base=0x2aabbb9ce0d8 > >query: me=1, them=PROC_NULL, size=4, disp=1, base=0x2aabbb9ce0d0 > >query: me=2, them=0, size=0, disp=1, base=(nil) > >query: me=2, them=1, size=4, disp=1, base=0x2b1579dd40d0 > >query: me=2, them=2, size=4, disp=1, base=0x2b1579dd40d4 > >query: me=2, them=3, size=4, disp=1, base=0x2b1579dd40d8 > >query: me=2, them=PROC_NULL, size=4, disp=1, base=0x2b1579dd40d0 > >query: me=3, them=0, size=0, disp=1, base=(nil) > >query: me=3, them=1, size=4, disp=1, base=0x2ac8d2c350d0 > >query: me=3, them=2, size=4, disp=1, base=0x2ac8d2c350d4 > >query: me=3, them=3, size=4, disp=1, base=0x2ac8d2c350d8 > >query: me=3, them=PROC_NULL, size=4, disp=1, base=0x2ac8d2c350d0 > > > > -- > > > > See attached. Output below. Note that the base you get for ranks 0 > > and > > 1 is the same, so you need to use the fact that size=0 at rank=0 to > > know > > not to dereference that pointer and expect to be writing into rank 0's > > memory, since you will write into rank 1's. > > I would probably add "if (size==0) base=NULL;" for good measure. > > Jeff > > > > $ mpirun -n 4 ./a.out > > > > query: me=0, them=0, size=0, disp=1, base=0x10bd64000 > > > > query: me=0, them=1, size=4, disp=1, base=0x10bd64000 > > > > query: me=0, them=2, size=4, disp=1, base=0x10bd64004 > > > > query: me=0, them=3, size=4, disp=1, base=0x10bd64008 > > > > query: me=0, them=PROC_NULL, size=4, disp=1, base=0x10bd64000 > > > > query: me=1, them=0, size=0, disp=1, base=0x102d3b000 > > > > query: me=1, them=1, size=4, disp=1, base=0x102d3b000 > > > > query: me=1, them=2, size=4, disp=1, base=0x102d3b004 > > > > query: me=1, them=3, size=4, disp=1, base=0x102d3b008 > > > > query: me=1, them=PROC_NULL, size=4, disp=1, base=0x102d3b000 > > > > query: me=2, them=0, size=0, disp=1, base=0x10aac1000 > > > > query: me=2, them=1, size=4, disp=1, base=0x10aac1000 > > > > query: me=2, them=2, size=4, disp=1, base=0x10aac1004 > > > > query: me=2, them=3, size=4, disp=1, base=0x10aac1008 > > > > query: me=2, them=PROC_NULL, size=4, disp=1, base=0x10aac1000 > > > > query: me=3, them=0, size=0, disp=1, base=0x100fa2000 > > > > query: me=3, them=1, si
Re: [OMPI users] shared memory zero size segment
Jeff probably ran with MPICH. Open MPI's are consistent with our choice of definition for size=0: query: me=1, them=0, size=0, disp=1, base=0x0 query: me=1, them=1, size=4, disp=1, base=0x1097e30f8 query: me=1, them=2, size=4, disp=1, base=0x1097e30fc query: me=1, them=3, size=4, disp=1, base=0x1097e3100 query: me=2, them=0, size=0, disp=1, base=0x0 query: me=2, them=1, size=4, disp=1, base=0x109fe10f8 query: me=2, them=2, size=4, disp=1, base=0x109fe10fc query: me=2, them=3, size=4, disp=1, base=0x109fe1100 query: me=2, them=PROC_NULL, size=4, disp=1, base=0x109fe10f8 query: me=3, them=0, size=0, disp=1, base=0x0 query: me=3, them=1, size=4, disp=1, base=0x1088f30f8 query: me=3, them=2, size=4, disp=1, base=0x1088f30fc query: me=3, them=3, size=4, disp=1, base=0x1088f3100 query: me=3, them=PROC_NULL, size=4, disp=1, base=0x1088f30f8 query: me=0, them=0, size=0, disp=1, base=0x0 query: me=0, them=1, size=4, disp=1, base=0x105b890f8 query: me=0, them=2, size=4, disp=1, base=0x105b890fc query: me=0, them=3, size=4, disp=1, base=0x105b89100 query: me=0, them=PROC_NULL, size=4, disp=1, base=0x105b890f8 query: me=1, them=PROC_NULL, size=4, disp=1, base=0x1097e30f8 To be portable only use MPI_Win_shared_query and do not rely on the return value of base if you pass size = 0. -Nathan On Thu, Feb 11, 2016 at 08:23:16PM +, Peter Wind wrote: >Thanks Jeff, that was an interesting result. The pointers are here well >defined, also for the zero size segment. >However I can't reproduce your output. I still get null pointers (output >below). >(I tried both 1.8.5 and 1.10.2 versions) >What could be the difference? >Peter >mpirun -np 4 a.out >query: me=0, them=0, size=0, disp=1, base=(nil) >query: me=0, them=1, size=4, disp=1, base=0x2aee280030d0 >query: me=0, them=2, size=4, disp=1, base=0x2aee280030d4 >query: me=0, them=3, size=4, disp=1, base=0x2aee280030d8 >query: me=0, them=PROC_NULL, size=4, disp=1, base=0x2aee280030d0 >query: me=1, them=0, size=0, disp=1, base=(nil) >query: me=1, them=1, size=4, disp=1, base=0x2aabbb9ce0d0 >query: me=1, them=2, size=4, disp=1, base=0x2aabbb9ce0d4 >query: me=1, them=3, size=4, disp=1, base=0x2aabbb9ce0d8 >query: me=1, them=PROC_NULL, size=4, disp=1, base=0x2aabbb9ce0d0 >query: me=2, them=0, size=0, disp=1, base=(nil) >query: me=2, them=1, size=4, disp=1, base=0x2b1579dd40d0 >query: me=2, them=2, size=4, disp=1, base=0x2b1579dd40d4 >query: me=2, them=3, size=4, disp=1, base=0x2b1579dd40d8 >query: me=2, them=PROC_NULL, size=4, disp=1, base=0x2b1579dd40d0 >query: me=3, them=0, size=0, disp=1, base=(nil) >query: me=3, them=1, size=4, disp=1, base=0x2ac8d2c350d0 >query: me=3, them=2, size=4, disp=1, base=0x2ac8d2c350d4 >query: me=3, them=3, size=4, disp=1, base=0x2ac8d2c350d8 >query: me=3, them=PROC_NULL, size=4, disp=1, base=0x2ac8d2c350d0 > > -- > > See attached. Output below. Note that the base you get for ranks 0 and > 1 is the same, so you need to use the fact that size=0 at rank=0 to know > not to dereference that pointer and expect to be writing into rank 0's > memory, since you will write into rank 1's. > I would probably add "if (size==0) base=NULL;" for good measure. > Jeff > > $ mpirun -n 4 ./a.out > > query: me=0, them=0, size=0, disp=1, base=0x10bd64000 > > query: me=0, them=1, size=4, disp=1, base=0x10bd64000 > > query: me=0, them=2, size=4, disp=1, base=0x10bd64004 > > query: me=0, them=3, size=4, disp=1, base=0x10bd64008 > > query: me=0, them=PROC_NULL, size=4, disp=1, base=0x10bd64000 > > query: me=1, them=0, size=0, disp=1, base=0x102d3b000 > > query: me=1, them=1, size=4, disp=1, base=0x102d3b000 > > query: me=1, them=2, size=4, disp=1, base=0x102d3b004 > > query: me=1, them=3, size=4, disp=1, base=0x102d3b008 > > query: me=1, them=PROC_NULL, size=4, disp=1, base=0x102d3b000 > > query: me=2, them=0, size=0, disp=1, base=0x10aac1000 > > query: me=2, them=1, size=4, disp=1, base=0x10aac1000 > > query: me=2, them=2, size=4, disp=1, base=0x10aac1004 > > query: me=2, them=3, size=4, disp=1, base=0x10aac1008 > > query: me=2, them=PROC_NULL, size=4, disp=1, base=0x10aac1000 > > query: me=3, them=0, size=0, disp=1, base=0x100fa2000 > > query: me=3, them=1, size=4, disp=1, base=0x100fa2000 > > query: me=3, them=2, size=4, disp=1, base=0x100fa2004 > > query: me=3, them=3, size=4, disp=1, base=0x100fa2008 > > query: me=3, them=PROC_NULL, size=4, disp=1, base=0x100fa2000 > > On Thu, Feb 11, 2016 at 8:55 AM, Jeff Hammond > wrote: > >On Thu, Feb 11, 2016 at 8:46 AM, Nathan Hjelm wrote: >> >> >> On Thu, Feb 11, 2016 at 02:17:40PM +, Peter Wind wrote: >> >I would add that the pre
Re: [OMPI users] shared memory zero size segment
You can be right semantically. But also the sentence "the first address in the memory segment of process i is consecutive with the last address in the memory segment of process i - 1" is not easy to interpret correctly for a zero size segment. There may be good reasons not to allocate the pointer for zero size segment. What I try to say is, that a new user reading the documentation, will not expect this behaviour before trying it out. Couldn't a small sentence in the documentation, like "the pointer should not be used for zero size segments" clarify this? Peter - Original Message - > > On Thu, Feb 11, 2016 at 02:17:40PM +, Peter Wind wrote: > >I would add that the present situation is bound to give problems for > >some > >users. > >It is natural to divide an array in segments, each process treating its > >own segment, but needing to read adjacent segments too. > >MPI_Win_allocate_shared seems to be designed for this. > >This will work fine as long as no segment as size zero. It can also be > >expected that most testing would be done with all segments larger than > >zero. > >The document adding "size = 0 is valid", would also make people > >confident > >that it will be consistent for that special case too. > > Nope, that statement says its ok for a rank to specify that the local > shared memory segment is 0 bytes. Nothing more. The standard > unfortunately does not define what pointer value is returned for a rank > that specifies size = 0. Not sure if the RMA working group intentionally > left that undefine... Anyway, Open MPI does not appear to be out of > compliance with the standard here. > > To be safe you should use MPI_Win_shared_query as suggested. You can > pass MPI_PROC_NULL as the rank to get the pointer for the first non-zero > sized segment in the shared memory window. > > -Nathan > HPC-5, LANL > > ___ > users mailing list > us...@open-mpi.org > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users > Link to this post: > http://www.open-mpi.org/community/lists/users/2016/02/28506.php
Re: [OMPI users] shared memory zero size segment
Thanks Jeff, that was an interesting result. The pointers are here well defined, also for the zero size segment. However I can't reproduce your output. I still get null pointers (output below). (I tried both 1.8.5 and 1.10.2 versions) What could be the difference? Peter mpirun -np 4 a.out query: me=0, them=0, size=0, disp=1, base=(nil) query: me=0, them=1, size=4, disp=1, base=0x2aee280030d0 query: me=0, them=2, size=4, disp=1, base=0x2aee280030d4 query: me=0, them=3, size=4, disp=1, base=0x2aee280030d8 query: me=0, them=PROC_NULL, size=4, disp=1, base=0x2aee280030d0 query: me=1, them=0, size=0, disp=1, base=(nil) query: me=1, them=1, size=4, disp=1, base=0x2aabbb9ce0d0 query: me=1, them=2, size=4, disp=1, base=0x2aabbb9ce0d4 query: me=1, them=3, size=4, disp=1, base=0x2aabbb9ce0d8 query: me=1, them=PROC_NULL, size=4, disp=1, base=0x2aabbb9ce0d0 query: me=2, them=0, size=0, disp=1, base=(nil) query: me=2, them=1, size=4, disp=1, base=0x2b1579dd40d0 query: me=2, them=2, size=4, disp=1, base=0x2b1579dd40d4 query: me=2, them=3, size=4, disp=1, base=0x2b1579dd40d8 query: me=2, them=PROC_NULL, size=4, disp=1, base=0x2b1579dd40d0 query: me=3, them=0, size=0, disp=1, base=(nil) query: me=3, them=1, size=4, disp=1, base=0x2ac8d2c350d0 query: me=3, them=2, size=4, disp=1, base=0x2ac8d2c350d4 query: me=3, them=3, size=4, disp=1, base=0x2ac8d2c350d8 query: me=3, them=PROC_NULL, size=4, disp=1, base=0x2ac8d2c350d0 - Original Message - > See attached. Output below. Note that the base you get for ranks 0 and 1 is > the same, so you need to use the fact that size=0 at rank=0 to know not to > dereference that pointer and expect to be writing into rank 0's memory, > since you will write into rank 1's. > I would probably add "if (size==0) base=NULL;" for good measure. > Jeff > $ mpirun -n 4 ./a.out > query: me=0, them=0, size=0, disp=1, base=0x10bd64000 > query: me=0, them=1, size=4, disp=1, base=0x10bd64000 > query: me=0, them=2, size=4, disp=1, base=0x10bd64004 > query: me=0, them=3, size=4, disp=1, base=0x10bd64008 > query: me=0, them=PROC_NULL, size=4, disp=1, base=0x10bd64000 > query: me=1, them=0, size=0, disp=1, base=0x102d3b000 > query: me=1, them=1, size=4, disp=1, base=0x102d3b000 > query: me=1, them=2, size=4, disp=1, base=0x102d3b004 > query: me=1, them=3, size=4, disp=1, base=0x102d3b008 > query: me=1, them=PROC_NULL, size=4, disp=1, base=0x102d3b000 > query: me=2, them=0, size=0, disp=1, base=0x10aac1000 > query: me=2, them=1, size=4, disp=1, base=0x10aac1000 > query: me=2, them=2, size=4, disp=1, base=0x10aac1004 > query: me=2, them=3, size=4, disp=1, base=0x10aac1008 > query: me=2, them=PROC_NULL, size=4, disp=1, base=0x10aac1000 > query: me=3, them=0, size=0, disp=1, base=0x100fa2000 > query: me=3, them=1, size=4, disp=1, base=0x100fa2000 > query: me=3, them=2, size=4, disp=1, base=0x100fa2004 > query: me=3, them=3, size=4, disp=1, base=0x100fa2008 > query: me=3, them=PROC_NULL, size=4, disp=1, base=0x100fa2000 > On Thu, Feb 11, 2016 at 8:55 AM, Jeff Hammond < jeff.scie...@gmail.com > > wrote: > > On Thu, Feb 11, 2016 at 8:46 AM, Nathan Hjelm < hje...@lanl.gov > wrote: > > > > > > > > > > > > On Thu, Feb 11, 2016 at 02:17:40PM +, Peter Wind wrote: > > > > > I would add that the present situation is bound to give problems for > > > > some > > > > > users. > > > > > It is natural to divide an array in segments, each process treating its > > > > > own segment, but needing to read adjacent segments too. > > > > > MPI_Win_allocate_shared seems to be designed for this. > > > > > This will work fine as long as no segment as size zero. It can also be > > > > > expected that most testing would be done with all segments larger than > > > > > zero. > > > > > The document adding "size = 0 is valid", would also make people > > > > confident > > > > > that it will be consistent for that special case too. > > > > > > > > Nope, that statement says its ok for a rank to specify that the local > > > > shared memory segment is 0 bytes. Nothing more. The standard > > > > unfortunately does not define what pointer value is returned for a rank > > > > that specifies size = 0. Not sure if the RMA working group intentionally > > > > left that undefine... Anyway, Open MPI does not appear to be out of > > > > compliance with the standard here. > > > > > > > MPI_Alloc_mem doesn't say what happens if you pass size=0 either. The RMA > > working group intentionally tries to maintain consistency with the rest of > > the MPI standard whenever possible, so we did not create a new semantic > > here. > > > MPI_Win_shared_query text includes this: > > > "If all processes in the group attached to the window specified size = 0, > > then the call returns size = 0 and a baseptr as if MPI_ALLOC_MEM was called > > with size = 0." > > > > > > > > To be safe you should use MPI_Win_shared_query as suggested. You can > > > > pass MPI_PROC_NULL as the rank
Re: [OMPI users] shared memory zero size segment
See attached. Output below. Note that the base you get for ranks 0 and 1 is the same, so you need to use the fact that size=0 at rank=0 to know not to dereference that pointer and expect to be writing into rank 0's memory, since you will write into rank 1's. I would probably add "if (size==0) base=NULL;" for good measure. Jeff $ mpirun -n 4 ./a.out query: me=0, them=0, size=0, disp=1, base=0x10bd64000 query: me=0, them=1, size=4, disp=1, base=0x10bd64000 query: me=0, them=2, size=4, disp=1, base=0x10bd64004 query: me=0, them=3, size=4, disp=1, base=0x10bd64008 query: me=0, them=PROC_NULL, size=4, disp=1, base=0x10bd64000 query: me=1, them=0, size=0, disp=1, base=0x102d3b000 query: me=1, them=1, size=4, disp=1, base=0x102d3b000 query: me=1, them=2, size=4, disp=1, base=0x102d3b004 query: me=1, them=3, size=4, disp=1, base=0x102d3b008 query: me=1, them=PROC_NULL, size=4, disp=1, base=0x102d3b000 query: me=2, them=0, size=0, disp=1, base=0x10aac1000 query: me=2, them=1, size=4, disp=1, base=0x10aac1000 query: me=2, them=2, size=4, disp=1, base=0x10aac1004 query: me=2, them=3, size=4, disp=1, base=0x10aac1008 query: me=2, them=PROC_NULL, size=4, disp=1, base=0x10aac1000 query: me=3, them=0, size=0, disp=1, base=0x100fa2000 query: me=3, them=1, size=4, disp=1, base=0x100fa2000 query: me=3, them=2, size=4, disp=1, base=0x100fa2004 query: me=3, them=3, size=4, disp=1, base=0x100fa2008 query: me=3, them=PROC_NULL, size=4, disp=1, base=0x100fa2000 On Thu, Feb 11, 2016 at 8:55 AM, Jeff Hammond wrote: > > > On Thu, Feb 11, 2016 at 8:46 AM, Nathan Hjelm wrote: > > > > > > On Thu, Feb 11, 2016 at 02:17:40PM +, Peter Wind wrote: > > >I would add that the present situation is bound to give problems > for some > > >users. > > >It is natural to divide an array in segments, each process treating > its > > >own segment, but needing to read adjacent segments too. > > >MPI_Win_allocate_shared seems to be designed for this. > > >This will work fine as long as no segment as size zero. It can also > be > > >expected that most testing would be done with all segments larger > than > > >zero. > > >The document adding "size = 0 is valid", would also make people > confident > > >that it will be consistent for that special case too. > > > > Nope, that statement says its ok for a rank to specify that the local > > shared memory segment is 0 bytes. Nothing more. The standard > > unfortunately does not define what pointer value is returned for a rank > > that specifies size = 0. Not sure if the RMA working group intentionally > > left that undefine... Anyway, Open MPI does not appear to be out of > > compliance with the standard here. > > > > MPI_Alloc_mem doesn't say what happens if you pass size=0 either. The RMA > working group intentionally tries to maintain consistency with the rest of > the MPI standard whenever possible, so we did not create a new semantic > here. > > MPI_Win_shared_query text includes this: > > "If all processes in the group attached to the window specified size = 0, > then the call returns size = 0 and a baseptr as if MPI_ALLOC_MEM was called > with size = 0." > > > > > To be safe you should use MPI_Win_shared_query as suggested. You can > > pass MPI_PROC_NULL as the rank to get the pointer for the first non-zero > > sized segment in the shared memory window. > > Indeed! I forgot about that. MPI_Win_shared_query solves this problem > for the user brilliantly. > > Jeff > > -- > Jeff Hammond > jeff.scie...@gmail.com > http://jeffhammond.github.io/ > -- Jeff Hammond jeff.scie...@gmail.com http://jeffhammond.github.io/ #include #include /* test zero size segment. run on at least 3 cpus mpirun -np 4 a.out */ int main(int argc, char** argv) { MPI_Init(NULL, NULL); int wsize, wrank; MPI_Comm_size(MPI_COMM_WORLD, &wsize); MPI_Comm_rank(MPI_COMM_WORLD, &wrank); MPI_Comm ncomm = MPI_COMM_NULL; MPI_Comm_split_type(MPI_COMM_WORLD, MPI_COMM_TYPE_SHARED, 0, MPI_INFO_NULL, &ncomm); MPI_Aint size = (wrank==0) ? 0 : sizeof(int); MPI_Win win = MPI_WIN_NULL; int * ptr = NULL; MPI_Win_allocate_shared(size, 1, MPI_INFO_NULL, MPI_COMM_WORLD, &ptr, &win); int nsize, nrank; MPI_Comm_size(MPI_COMM_WORLD, &nsize); MPI_Comm_rank(MPI_COMM_WORLD, &nrank); for (int r=0; r
Re: [OMPI users] shared memory zero size segment
On Thu, Feb 11, 2016 at 8:46 AM, Nathan Hjelm wrote: > > > On Thu, Feb 11, 2016 at 02:17:40PM +, Peter Wind wrote: > >I would add that the present situation is bound to give problems for some > >users. > >It is natural to divide an array in segments, each process treating its > >own segment, but needing to read adjacent segments too. > >MPI_Win_allocate_shared seems to be designed for this. > >This will work fine as long as no segment as size zero. It can also be > >expected that most testing would be done with all segments larger than > >zero. > >The document adding "size = 0 is valid", would also make people confident > >that it will be consistent for that special case too. > > Nope, that statement says its ok for a rank to specify that the local > shared memory segment is 0 bytes. Nothing more. The standard > unfortunately does not define what pointer value is returned for a rank > that specifies size = 0. Not sure if the RMA working group intentionally > left that undefine... Anyway, Open MPI does not appear to be out of > compliance with the standard here. > MPI_Alloc_mem doesn't say what happens if you pass size=0 either. The RMA working group intentionally tries to maintain consistency with the rest of the MPI standard whenever possible, so we did not create a new semantic here. MPI_Win_shared_query text includes this: "If all processes in the group attached to the window specified size = 0, then the call returns size = 0 and a baseptr as if MPI_ALLOC_MEM was called with size = 0." > > To be safe you should use MPI_Win_shared_query as suggested. You can > pass MPI_PROC_NULL as the rank to get the pointer for the first non-zero > sized segment in the shared memory window. Indeed! I forgot about that. MPI_Win_shared_query solves this problem for the user brilliantly. Jeff -- Jeff Hammond jeff.scie...@gmail.com http://jeffhammond.github.io/
Re: [OMPI users] shared memory zero size segment
On Thu, Feb 11, 2016 at 02:17:40PM +, Peter Wind wrote: >I would add that the present situation is bound to give problems for some >users. >It is natural to divide an array in segments, each process treating its >own segment, but needing to read adjacent segments too. >MPI_Win_allocate_shared seems to be designed for this. >This will work fine as long as no segment as size zero. It can also be >expected that most testing would be done with all segments larger than >zero. >The document adding "size = 0 is valid", would also make people confident >that it will be consistent for that special case too. Nope, that statement says its ok for a rank to specify that the local shared memory segment is 0 bytes. Nothing more. The standard unfortunately does not define what pointer value is returned for a rank that specifies size = 0. Not sure if the RMA working group intentionally left that undefine... Anyway, Open MPI does not appear to be out of compliance with the standard here. To be safe you should use MPI_Win_shared_query as suggested. You can pass MPI_PROC_NULL as the rank to get the pointer for the first non-zero sized segment in the shared memory window. -Nathan HPC-5, LANL pgp4wih319kzq.pgp Description: PGP signature
Re: [OMPI users] shared memory zero size segment
I would add that the present situation is bound to give problems for some users. It is natural to divide an array in segments, each process treating its own segment, but needing to read adjacent segments too. MPI_Win_allocate_shared seems to be designed for this. This will work fine as long as no segment as size zero. It can also be expected that most testing would be done with all segments larger than zero. The document adding "size = 0 is valid", would also make people confident that it will be consistent for that special case too. Then long down the road of the development of a particular code some special case will use a segment of size zero, and it will be hard to trace back this error to the mpi library. Peter - Original Message - Yes, that is what I meant. Enclosed is a C example. The point is that the code would logically make sense for task 0, but since it asks for a segment of size=0, it only gets a null pointer, which cannot be used to access the shared parts. Peter - Original Message - I think Peter's point is that if - the windows uses contiguous memory *and* - all tasks knows how much memory was allocated by all other tasks in the window then it could/should be possible to get rid of MPI_Win_shared_query that is likely true if no task allocates zero byte. now, if a task allocates zero byte, MPI_Win_allocate_shared could return a null pointer and hence makes MPI_Win_shared_query usage mandatory. in his example, task 0 allocates zero bytes, so he was expecting the returned pointer on task zero points to the memory allocated by task 1. if "may enable" should be read as "does enable", then returning a null pointer can be seen as a bug. if "may enable" can be read as "does not always enable", the returning a null pointer is compliant with the standard. I am clearly not good at reading/interpreting the standard, so using MPI_Win_shared_query is my recommended way to get it work. (feel free to call it "bulletproof", "overkill", or even "right") Cheers, Gilles On Thursday, February 11, 2016, Jeff Hammond < jeff.scie...@gmail.com > wrote: On Wed, Feb 10, 2016 at 8:44 AM, Peter Wind < peter.w...@met.no > wrote: I agree that in practice the best practice would be to use Win_shared_query. Still I am confused by this part in the documentation: "The allocated memory is contiguous across process ranks unless the info key alloc_shared_noncontig is specified. Contiguous across process ranks means that the first address in the memory segment of process i is consecutive with the last address in the memory segment of process i - 1. This may enable the user to calculate remote address offsets with local information only." Isn't this an encouragement to use the pointer of Win_allocate_shared directly? No, it is not. Win_allocate_shared only gives you the pointer to the portion of the allocation that is owned by the calling process. If you want to access the whole slab, call Win_shared_query(..,rank=0,..) and use the resulting baseptr. I attempted to modify your code to be more correct, but I don't know enough Fortran to get it right. If you can parse C examples, I'll provide some of those. Jeff Peter I don't know about bulletproof, but Win_shared_query is the *only* valid way to get the addresses of memory in other processes associated with a window. The default for Win_allocate_shared is contiguous memory, but it can and likely will be mapped differently into each process, in which case only relative offsets are transferrable. Jeff On Wed, Feb 10, 2016 at 4:19 AM, Gilles Gouaillardet < gilles.gouaillar...@gmail.com > wrote: Peter, The bulletproof way is to use MPI_Win_shared_query after MPI_Win_allocate_shared. I do not know if current behavior is a bug or a feature... Cheers, Gilles On Wednesday, February 10, 2016, Peter Wind < peter.w...@met.no > wrote: Hi, Under fortran, MPI_Win_allocate_shared is called with a window size of zero for some processes. The output pointer is then not valid for these processes (null pointer). Did I understood this wrongly? shouldn't the pointers be contiguous, so that for a zero sized window, the pointer should point to the start of the segment of the next rank? The documentation explicitly specifies "size = 0 is valid". Attached a small code, where rank=0 allocate a window of size zero. All the other ranks get valid pointers, except rank 0. Best regards, Peter ___ users mailing list us...@open-mpi.org Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users Link to this post: http://www.open-mpi.org/community/lists/users/2016/02/28485.php ___ users mailing list us...@open-mpi.org Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users Link to this post: http://www.open-mpi.org/community/lists/users/2016/0
Re: [OMPI users] shared memory zero size segment
Yes, that is what I meant. Enclosed is a C example. The point is that the code would logically make sense for task 0, but since it asks for a segment of size=0, it only gets a null pointer, which cannot be used to access the shared parts. Peter - Original Message - > I think Peter's point is that if > - the windows uses contiguous memory > *and* > - all tasks knows how much memory was allocated by all other tasks in the > window > then it could/should be possible to get rid of MPI_Win_shared_query > that is likely true if no task allocates zero byte. > now, if a task allocates zero byte, MPI_Win_allocate_shared could return a > null pointer and hence makes MPI_Win_shared_query usage mandatory. > in his example, task 0 allocates zero bytes, so he was expecting the returned > pointer on task zero points to the memory allocated by task 1. > if "may enable" should be read as "does enable", then returning a null > pointer can be seen as a bug. > if "may enable" can be read as "does not always enable", the returning a null > pointer is compliant with the standard. > I am clearly not good at reading/interpreting the standard, so using > MPI_Win_shared_query is my recommended way to get it work. > (feel free to call it "bulletproof", "overkill", or even "right") > Cheers, > Gilles > On Thursday, February 11, 2016, Jeff Hammond < jeff.scie...@gmail.com > > wrote: > > On Wed, Feb 10, 2016 at 8:44 AM, Peter Wind < peter.w...@met.no > wrote: > > > > I agree that in practice the best practice would be to use > > > Win_shared_query. > > > > > > Still I am confused by this part in the documentation: > > > > > > "The allocated memory is contiguous across process ranks unless the info > > > key > > > alloc_shared_noncontig is specified. Contiguous across process ranks > > > means > > > that the first address in the memory segment of process i is consecutive > > > with the last address in the memory segment of process i - 1. This may > > > enable the user to calculate remote address offsets with local > > > information > > > only." > > > > > > Isn't this an encouragement to use the pointer of Win_allocate_shared > > > directly? > > > > > No, it is not. Win_allocate_shared only gives you the pointer to the > > portion > > of the allocation that is owned by the calling process. If you want to > > access the whole slab, call Win_shared_query(..,rank=0,..) and use the > > resulting baseptr. > > > I attempted to modify your code to be more correct, but I don't know enough > > Fortran to get it right. If you can parse C examples, I'll provide some of > > those. > > > Jeff > > > > Peter > > > > > > > I don't know about bulletproof, but Win_shared_query is the *only* > > > > valid > > > > way > > > > to get the addresses of memory in other processes associated with a > > > > window. > > > > > > > > > > The default for Win_allocate_shared is contiguous memory, but it can > > > > and > > > > likely will be mapped differently into each process, in which case only > > > > relative offsets are transferrable. > > > > > > > > > > Jeff > > > > > > > > > > On Wed, Feb 10, 2016 at 4:19 AM, Gilles Gouaillardet < > > > > gilles.gouaillar...@gmail.com > wrote: > > > > > > > > > > > Peter, > > > > > > > > > > > > > > > The bulletproof way is to use MPI_Win_shared_query after > > > > > MPI_Win_allocate_shared. > > > > > > > > > > > > > > > I do not know if current behavior is a bug or a feature... > > > > > > > > > > > > > > > Cheers, > > > > > > > > > > > > > > > Gilles > > > > > > > > > > > > > > > On Wednesday, February 10, 2016, Peter Wind < peter.w...@met.no > > > > > > wrote: > > > > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > > > > > > > > Under fortran, MPI_Win_allocate_shared is called with a window size > > > > > > of > > > > > > zero > > > > > > for some processes. > > > > > > > > > > > > > > > > > > > > > The output pointer is then not valid for these processes (null > > > > > > pointer). > > > > > > > > > > > > > > > > > > > > > Did I understood this wrongly? shouldn't the pointers be > > > > > > contiguous, > > > > > > so > > > > > > that > > > > > > for a zero sized window, the pointer should point to the start of > > > > > > the > > > > > > segment of the next rank? > > > > > > > > > > > > > > > > > > > > > The documentation explicitly specifies "size = 0 is valid". > > > > > > > > > > > > > > > > > > > > > Attached a small code, where rank=0 allocate a window of size zero. > > > > > > All > > > > > > the > > > > > > other ranks get valid pointers, except rank 0. > > > > > > > > > > > > > > > > > > > > > Best regards, > > > > > > > > > > > > > > > > > > > > > Peter > > > > > > > > > > > > > > > > > > > > > ___ > > > > > > > > > > > > > > > > > > > > > users mailing list > > > > > > > > > > > > > > > > > > > > > us...@open-mpi.org >
Re: [OMPI users] shared memory zero size segment
I think Peter's point is that if - the windows uses contiguous memory *and* - all tasks knows how much memory was allocated by all other tasks in the window then it could/should be possible to get rid of MPI_Win_shared_query that is likely true if no task allocates zero byte. now, if a task allocates zero byte, MPI_Win_allocate_shared could return a null pointer and hence makes MPI_Win_shared_query usage mandatory. in his example, task 0 allocates zero bytes, so he was expecting the returned pointer on task zero points to the memory allocated by task 1. if "may enable" should be read as "does enable", then returning a null pointer can be seen as a bug. if "may enable" can be read as "does not always enable", the returning a null pointer is compliant with the standard. I am clearly not good at reading/interpreting the standard, so using MPI_Win_shared_query is my recommended way to get it work. (feel free to call it "bulletproof", "overkill", or even "right") Cheers, Gilles On Thursday, February 11, 2016, Jeff Hammond wrote: > > > On Wed, Feb 10, 2016 at 8:44 AM, Peter Wind > wrote: > >> I agree that in practice the best practice would be to use >> Win_shared_query. >> >> Still I am confused by this part in the documentation: >> "The allocated memory is contiguous across process ranks unless the info >> key *alloc_shared_noncontig* is specified. Contiguous across process >> ranks means that the first address in the memory segment of process i is >> consecutive with the last address in the memory segment of process i - 1. >> This may enable the user to calculate remote address offsets with local >> information only." >> >> Isn't this an encouragement to use the pointer of Win_allocate_shared >> directly? >> >> > No, it is not. Win_allocate_shared only gives you the pointer to the > portion of the allocation that is owned by the calling process. If you > want to access the whole slab, call Win_shared_query(..,rank=0,..) and use > the resulting baseptr. > > I attempted to modify your code to be more correct, but I don't know > enough Fortran to get it right. If you can parse C examples, I'll provide > some of those. > > Jeff > > >> Peter >> >> -- >> >> I don't know about bulletproof, but Win_shared_query is the *only* valid >> way to get the addresses of memory in other processes associated with a >> window. >> >> The default for Win_allocate_shared is contiguous memory, but it can and >> likely will be mapped differently into each process, in which case only >> relative offsets are transferrable. >> >> Jeff >> >> On Wed, Feb 10, 2016 at 4:19 AM, Gilles Gouaillardet < >> gilles.gouaillar...@gmail.com >> > wrote: >> >>> Peter, >>> >>> The bulletproof way is to use MPI_Win_shared_query after >>> MPI_Win_allocate_shared. >>> I do not know if current behavior is a bug or a feature... >>> >>> Cheers, >>> >>> Gilles >>> >>> >>> On Wednesday, February 10, 2016, Peter Wind >> > wrote: >>> Hi, Under fortran, MPI_Win_allocate_shared is called with a window size of zero for some processes. The output pointer is then not valid for these processes (null pointer). Did I understood this wrongly? shouldn't the pointers be contiguous, so that for a zero sized window, the pointer should point to the start of the segment of the next rank? The documentation explicitly specifies "size = 0 is valid". Attached a small code, where rank=0 allocate a window of size zero. All the other ranks get valid pointers, except rank 0. Best regards, Peter ___ users mailing list us...@open-mpi.org Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users Link to this post: http://www.open-mpi.org/community/lists/users/2016/02/28485.php >>> >>> ___ >>> users mailing list >>> us...@open-mpi.org >>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users >>> Link to this post: >>> http://www.open-mpi.org/community/lists/users/2016/02/28493.php >>> >> >> >> >> -- >> Jeff Hammond >> jeff.scie...@gmail.com >> >> http://jeffhammond.github.io/ >> >> ___ >> users mailing list >> us...@open-mpi.org >> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users >> Link to this post: >> http://www.open-mpi.org/community/lists/users/2016/02/28496.php >> >> >> >> ___ >> users mailing list >> us...@open-mpi.org >> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users >> Link to this post: >> http://www.open-mpi.org/community/lists/users/2016/02/28497.php >> > > > > -- > Jeff Hammond > jeff.scie...@gmail.com > > http://jeffhammond.github.io/ >
Re: [OMPI users] shared memory zero size segment
On Wed, Feb 10, 2016 at 8:44 AM, Peter Wind wrote: > I agree that in practice the best practice would be to use > Win_shared_query. > > Still I am confused by this part in the documentation: > "The allocated memory is contiguous across process ranks unless the info > key *alloc_shared_noncontig* is specified. Contiguous across process > ranks means that the first address in the memory segment of process i is > consecutive with the last address in the memory segment of process i - 1. > This may enable the user to calculate remote address offsets with local > information only." > > Isn't this an encouragement to use the pointer of Win_allocate_shared > directly? > > No, it is not. Win_allocate_shared only gives you the pointer to the portion of the allocation that is owned by the calling process. If you want to access the whole slab, call Win_shared_query(..,rank=0,..) and use the resulting baseptr. I attempted to modify your code to be more correct, but I don't know enough Fortran to get it right. If you can parse C examples, I'll provide some of those. Jeff > Peter > > -- > > I don't know about bulletproof, but Win_shared_query is the *only* valid > way to get the addresses of memory in other processes associated with a > window. > > The default for Win_allocate_shared is contiguous memory, but it can and > likely will be mapped differently into each process, in which case only > relative offsets are transferrable. > > Jeff > > On Wed, Feb 10, 2016 at 4:19 AM, Gilles Gouaillardet < > gilles.gouaillar...@gmail.com> wrote: > >> Peter, >> >> The bulletproof way is to use MPI_Win_shared_query after >> MPI_Win_allocate_shared. >> I do not know if current behavior is a bug or a feature... >> >> Cheers, >> >> Gilles >> >> >> On Wednesday, February 10, 2016, Peter Wind wrote: >> >>> Hi, >>> >>> Under fortran, MPI_Win_allocate_shared is called with a window size of >>> zero for some processes. >>> The output pointer is then not valid for these processes (null pointer). >>> Did I understood this wrongly? shouldn't the pointers be contiguous, so >>> that for a zero sized window, the pointer should point to the start of the >>> segment of the next rank? >>> The documentation explicitly specifies "size = 0 is valid". >>> >>> Attached a small code, where rank=0 allocate a window of size zero. All >>> the other ranks get valid pointers, except rank 0. >>> >>> Best regards, >>> Peter >>> ___ >>> users mailing list >>> us...@open-mpi.org >>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users >>> Link to this post: >>> http://www.open-mpi.org/community/lists/users/2016/02/28485.php >>> >> >> ___ >> users mailing list >> us...@open-mpi.org >> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users >> Link to this post: >> http://www.open-mpi.org/community/lists/users/2016/02/28493.php >> > > > > -- > Jeff Hammond > jeff.scie...@gmail.com > http://jeffhammond.github.io/ > > ___ > users mailing list > us...@open-mpi.org > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users > Link to this post: > http://www.open-mpi.org/community/lists/users/2016/02/28496.php > > > > ___ > users mailing list > us...@open-mpi.org > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users > Link to this post: > http://www.open-mpi.org/community/lists/users/2016/02/28497.php > -- Jeff Hammond jeff.scie...@gmail.com http://jeffhammond.github.io/ sharetest.f90 Description: Binary data
Re: [OMPI users] shared memory zero size segment
I agree that in practice the best practice would be to use Win_shared_query. Still I am confused by this part in the documentation: "The allocated memory is contiguous across process ranks unless the info key alloc_shared_noncontig is specified. Contiguous across process ranks means that the first address in the memory segment of process i is consecutive with the last address in the memory segment of process i - 1. This may enable the user to calculate remote address offsets with local information only." Isn't this an encouragement to use the pointer of Win_allocate_shared directly? Peter - Original Message - > I don't know about bulletproof, but Win_shared_query is the *only* valid way > to get the addresses of memory in other processes associated with a window. > The default for Win_allocate_shared is contiguous memory, but it can and > likely will be mapped differently into each process, in which case only > relative offsets are transferrable. > Jeff > On Wed, Feb 10, 2016 at 4:19 AM, Gilles Gouaillardet < > gilles.gouaillar...@gmail.com > wrote: > > Peter, > > > The bulletproof way is to use MPI_Win_shared_query after > > MPI_Win_allocate_shared. > > > I do not know if current behavior is a bug or a feature... > > > Cheers, > > > Gilles > > > On Wednesday, February 10, 2016, Peter Wind < peter.w...@met.no > wrote: > > > > Hi, > > > > > > Under fortran, MPI_Win_allocate_shared is called with a window size of > > > zero > > > for some processes. > > > > > > The output pointer is then not valid for these processes (null pointer). > > > > > > Did I understood this wrongly? shouldn't the pointers be contiguous, so > > > that > > > for a zero sized window, the pointer should point to the start of the > > > segment of the next rank? > > > > > > The documentation explicitly specifies "size = 0 is valid". > > > > > > Attached a small code, where rank=0 allocate a window of size zero. All > > > the > > > other ranks get valid pointers, except rank 0. > > > > > > Best regards, > > > > > > Peter > > > > > > ___ > > > > > > users mailing list > > > > > > us...@open-mpi.org > > > > > > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users > > > > > > Link to this post: > > > http://www.open-mpi.org/community/lists/users/2016/02/28485.php > > > > > ___ > > > users mailing list > > > us...@open-mpi.org > > > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users > > > Link to this post: > > http://www.open-mpi.org/community/lists/users/2016/02/28493.php > > -- > Jeff Hammond > jeff.scie...@gmail.com > http://jeffhammond.github.io/ > ___ > users mailing list > us...@open-mpi.org > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users > Link to this post: > http://www.open-mpi.org/community/lists/users/2016/02/28496.php
Re: [OMPI users] shared memory zero size segment
I don't know about bulletproof, but Win_shared_query is the *only* valid way to get the addresses of memory in other processes associated with a window. The default for Win_allocate_shared is contiguous memory, but it can and likely will be mapped differently into each process, in which case only relative offsets are transferrable. Jeff On Wed, Feb 10, 2016 at 4:19 AM, Gilles Gouaillardet < gilles.gouaillar...@gmail.com> wrote: > Peter, > > The bulletproof way is to use MPI_Win_shared_query after > MPI_Win_allocate_shared. > I do not know if current behavior is a bug or a feature... > > Cheers, > > Gilles > > > On Wednesday, February 10, 2016, Peter Wind wrote: > >> Hi, >> >> Under fortran, MPI_Win_allocate_shared is called with a window size of >> zero for some processes. >> The output pointer is then not valid for these processes (null pointer). >> Did I understood this wrongly? shouldn't the pointers be contiguous, so >> that for a zero sized window, the pointer should point to the start of the >> segment of the next rank? >> The documentation explicitly specifies "size = 0 is valid". >> >> Attached a small code, where rank=0 allocate a window of size zero. All >> the other ranks get valid pointers, except rank 0. >> >> Best regards, >> Peter >> ___ >> users mailing list >> us...@open-mpi.org >> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users >> Link to this post: >> http://www.open-mpi.org/community/lists/users/2016/02/28485.php >> > > ___ > users mailing list > us...@open-mpi.org > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users > Link to this post: > http://www.open-mpi.org/community/lists/users/2016/02/28493.php > -- Jeff Hammond jeff.scie...@gmail.com http://jeffhammond.github.io/
Re: [OMPI users] shared memory zero size segment
Peter, The bulletproof way is to use MPI_Win_shared_query after MPI_Win_allocate_shared. I do not know if current behavior is a bug or a feature... Cheers, Gilles On Wednesday, February 10, 2016, Peter Wind wrote: > Hi, > > Under fortran, MPI_Win_allocate_shared is called with a window size of > zero for some processes. > The output pointer is then not valid for these processes (null pointer). > Did I understood this wrongly? shouldn't the pointers be contiguous, so > that for a zero sized window, the pointer should point to the start of the > segment of the next rank? > The documentation explicitly specifies "size = 0 is valid". > > Attached a small code, where rank=0 allocate a window of size zero. All > the other ranks get valid pointers, except rank 0. > > Best regards, > Peter > ___ > users mailing list > us...@open-mpi.org > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users > Link to this post: > http://www.open-mpi.org/community/lists/users/2016/02/28485.php >
Re: [OMPI users] shared memory zero size segment
Sorry for that, here is the attachement! Peter - Original Message - > Peter -- > > Somewhere along the way, your attachment got lost. Could you re-send? > > Thanks. > > > > On Feb 10, 2016, at 5:56 AM, Peter Wind wrote: > > > > Hi, > > > > Under fortran, MPI_Win_allocate_shared is called with a window size of zero > > for some processes. > > The output pointer is then not valid for these processes (null pointer). > > Did I understood this wrongly? shouldn't the pointers be contiguous, so > > that for a zero sized window, the pointer should point to the start of the > > segment of the next rank? > > The documentation explicitly specifies "size = 0 is valid". > > > > Attached a small code, where rank=0 allocate a window of size zero. All the > > other ranks get valid pointers, except rank 0. > > > > Best regards, > > Peter > > ___ > > users mailing list > > us...@open-mpi.org > > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users > > Link to this post: > > http://www.open-mpi.org/community/lists/users/2016/02/28485.php > > > -- > Jeff Squyres > jsquy...@cisco.com > For corporate legal information go to: > http://www.cisco.com/web/about/doing_business/legal/cri/ > > ___ > users mailing list > us...@open-mpi.org > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users > Link to this post: > http://www.open-mpi.org/community/lists/users/2016/02/28486.php > program sharetest ! test zero size segment. ! run on at least 3 cpus ! mpirun -np 4 a.out use mpi use, intrinsic :: iso_c_binding, implicit none integer, parameter :: nsize = 20 integer, pointer :: array(:) integer:: num_procs integer:: ierr integer:: irank, irank_group integer:: win integer:: disp_unit type(c_ptr):: cp1 type(c_ptr):: cp2 integer(MPI_ADDRESS_KIND) :: win_size integer(MPI_ADDRESS_KIND) :: segment_size call MPI_Init(ierr) call MPI_Comm_size(MPI_COMM_WORLD, num_procs, ierr) call MPI_Comm_rank(MPI_COMM_WORLD, irank, ierr) disp_unit = sizeof(1) win_size = irank*disp_unit call MPI_Win_allocate_shared(win_size, disp_unit, MPI_INFO_NULL, MPI_COMM_WORLD, cp1, win, ierr) ! write(*,*)'rank ', irank,', pointer ',cp1 call c_f_pointer(cp1, array, [nsize]) 77 format(4(A,I3)) if(irank/=0)then array(1)=irank CALL MPI_BARRIER(MPI_COMM_WORLD, ierr) if(irank/=num_procs-1)then print 77, ' rank', irank, ': array(1)', array(1),' shared with next rank: ',array(irank+1) else print 77, ' rank', irank, ': array(1)', array(1),' shared with previous rank: ',array(0) endif CALL MPI_BARRIER(MPI_COMM_WORLD, ierr) else CALL MPI_BARRIER(MPI_COMM_WORLD, ierr) CALL MPI_BARRIER(MPI_COMM_WORLD, ierr) if(.not.associated(array))then print 77, 'zero pointer found, rank', irank else print 77, ' rank', irank, ' array associated ' print 77, ' rank', irank, ': array(1) ', array(1),' shared with next rank: ',array(irank+1) endif endif call MPI_Finalize(ierr) end program sharetest
Re: [OMPI users] shared memory zero size segment
- Original Message - > Peter -- > > Somewhere along the way, your attachment got lost. Could you re-send? > > Thanks. > > > > On Feb 10, 2016, at 5:56 AM, Peter Wind wrote: > > > > Hi, > > > > Under fortran, MPI_Win_allocate_shared is called with a window size of zero > > for some processes. > > The output pointer is then not valid for these processes (null pointer). > > Did I understood this wrongly? shouldn't the pointers be contiguous, so > > that for a zero sized window, the pointer should point to the start of the > > segment of the next rank? > > The documentation explicitly specifies "size = 0 is valid". > > > > Attached a small code, where rank=0 allocate a window of size zero. All the > > other ranks get valid pointers, except rank 0. > > > > Best regards, > > Peter > > ___ > > users mailing list > > us...@open-mpi.org > > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users > > Link to this post: > > http://www.open-mpi.org/community/lists/users/2016/02/28485.php > > > -- > Jeff Squyres > jsquy...@cisco.com > For corporate legal information go to: > http://www.cisco.com/web/about/doing_business/legal/cri/ > > ___ > users mailing list > us...@open-mpi.org > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users > Link to this post: > http://www.open-mpi.org/community/lists/users/2016/02/28486.php > program sharetest ! test zero size segment. ! run on at least 3 cpus ! mpirun -np 4 a.out use mpi use, intrinsic :: iso_c_binding, implicit none integer, parameter :: nsize = 20 integer, pointer :: array(:) integer:: num_procs integer:: ierr integer:: irank, irank_group integer:: win integer:: disp_unit type(c_ptr):: cp1 type(c_ptr):: cp2 integer(MPI_ADDRESS_KIND) :: win_size integer(MPI_ADDRESS_KIND) :: segment_size call MPI_Init(ierr) call MPI_Comm_size(MPI_COMM_WORLD, num_procs, ierr) call MPI_Comm_rank(MPI_COMM_WORLD, irank, ierr) disp_unit = sizeof(1) win_size = irank*disp_unit call MPI_Win_allocate_shared(win_size, disp_unit, MPI_INFO_NULL, MPI_COMM_WORLD, cp1, win, ierr) ! write(*,*)'rank ', irank,', pointer ',cp1 call c_f_pointer(cp1, array, [nsize]) 77 format(4(A,I3)) if(irank/=0)then array(1)=irank CALL MPI_BARRIER(MPI_COMM_WORLD, ierr) if(irank/=num_procs-1)then print 77, ' rank', irank, ': array(1)', array(1),' shared with next rank: ',array(irank+1) else print 77, ' rank', irank, ': array(1)', array(1),' shared with previous rank: ',array(0) endif CALL MPI_BARRIER(MPI_COMM_WORLD, ierr) else CALL MPI_BARRIER(MPI_COMM_WORLD, ierr) CALL MPI_BARRIER(MPI_COMM_WORLD, ierr) if(.not.associated(array))then print 77, 'zero pointer found, rank', irank else print 77, ' rank', irank, ' array associated ' print 77, ' rank', irank, ': array(1) ', array(1),' shared with next rank: ',array(irank+1) endif endif call MPI_Finalize(ierr) end program sharetest
Re: [OMPI users] shared memory zero size segment
Peter -- Somewhere along the way, your attachment got lost. Could you re-send? Thanks. > On Feb 10, 2016, at 5:56 AM, Peter Wind wrote: > > Hi, > > Under fortran, MPI_Win_allocate_shared is called with a window size of zero > for some processes. > The output pointer is then not valid for these processes (null pointer). > Did I understood this wrongly? shouldn't the pointers be contiguous, so that > for a zero sized window, the pointer should point to the start of the segment > of the next rank? > The documentation explicitly specifies "size = 0 is valid". > > Attached a small code, where rank=0 allocate a window of size zero. All the > other ranks get valid pointers, except rank 0. > > Best regards, > Peter > ___ > users mailing list > us...@open-mpi.org > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users > Link to this post: > http://www.open-mpi.org/community/lists/users/2016/02/28485.php -- Jeff Squyres jsquy...@cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/