Re: [OMPI users] MPI_IN_PLACE

[Gilles Gouaillardet]

Thanks Jeff for your offer, i will contact you off-list later


i tried a gcc+gfortran and gcc+ifort on both linux and OS X
so far, only gcc+ifort on OS X is failing
i will try icc+ifort on OS X from now

short story, MPI_IN_PLACE is not recognized as such by the ompi
fortran wrapper, and i do not know why.

the attached program can be used to evidence the issue.


Cheers,

Gilles

On Thu, Jul 27, 2017 at 2:15 PM, Volker Blum  wrote:
> Thanks! That’s great. Sounds like the exact combination I have here.
>
> Thanks also to George. Sorry that the test did not trigger on a more standard 
> platform - that would have simplified things.
>
> Best wishes
> Volker
>
>> On Jul 27, 2017, at 3:56 AM, Gilles Gouaillardet  wrote:
>>
>> Folks,
>>
>>
>> I am able to reproduce the issue on OS X (Sierra) with stock gcc (aka clang) 
>> and ifort 17.0.4
>>
>>
>> i will investigate this from now
>>
>>
>> Cheers,
>>
>> Gilles
>>
>> On 7/27/2017 9:28 AM, George Bosilca wrote:
>>> Volker,
>>>
>>> Unfortunately, I can't replicate with icc. I tried on a x86_64 box with 
>>> Intel compiler chain 17.0.4 20170411 to no avail. I also tested the 
>>> 3.0.0-rc1 tarball and the current master, and you test completes without 
>>> errors on all cases.
>>>
>>> Once you figure out an environment where you can consistently replicate the 
>>> issue, I would suggest to attach to the processes and:
>>> - make sure the MPI_IN_PLACE as seen through the Fortran layer matches what 
>>> the C layer expects
>>> - what is the collective algorithm used by Open MPI
>>>
>>> I have a "Fortran 101" level question. When you pass an array a(:) as 
>>> argument, what exactly gets passed via the Fortran interface to the 
>>> corresponding C function ?
>>>
>>>  George.
>>>
>>> On Wed, Jul 26, 2017 at 1:55 PM, Volker Blum >> > wrote:
>>>
>>>Thanks! Yes, trying with Intel 2017 would be very nice.
>>>
>>>> On Jul 26, 2017, at 6:12 PM, George Bosilca >>> wrote:
>>>>
>>>> No, I don't have (or used where they were available) the Intel
>>>compiler. I used clang and gfortran. I can try on a Linux box with
>>>the Intel 2017 compilers.
>>>>
>>>>   George.
>>>>
>>>>
>>>>
>>>> On Wed, Jul 26, 2017 at 11:59 AM, Volker Blum
>>>> wrote:
>>>> Did you use Intel Fortran 2017 as well?
>>>>
>>>> (I’m asking because I did see the same issue with a combination
>>>of an earlier Intel Fortran 2017 version and OpenMPI on an
>>>Intel/Infiniband Linux HPC machine … but not Intel Fortran 2016 on
>>>the same machine. Perhaps I can revive my access to that
>>>combination somehow.)
>>>>
>>>> Best wishes
>>>> Volker
>>>>
>>>> > On Jul 26, 2017, at 5:55 PM, George Bosilca
>>>> wrote:
>>>> >
>>>> > I thought that maybe the underlying allreduce algorithm fails
>>>to support MPI_IN_PLACE correctly, but I can't replicate on any
>>>machine (including OSX) with any number of processes.
>>>> >
>>>> >   George.
>>>> >
>>>> >
>>>> >
>>>> > On Wed, Jul 26, 2017 at 10:59 AM, Volker Blum
>>>> wrote:
>>>> > Thanks!
>>>> >
>>>> > I tried ‘use mpi’, which compiles fine.
>>>> >
>>>> > Same result as with ‘include mpif.h', in that the output is
>>>> >
>>>> >  * MPI_IN_PLACE does not appear to work as intended.
>>>> >  * Checking whether MPI_ALLREDUCE works at all.
>>>> >  * Without MPI_IN_PLACE, MPI_ALLREDUCE appears to work.
>>>> >
>>>> > Hm. Any other thoughts?
>>>> >
>>>> > Thanks again!
>>>> > Best wishes
>>>> > Volker
>>>> >
>>>> > > On Jul 26, 2017, at 4:06 PM, Gilles Gouaillardet
>>>>>> wrote:
>>>> > >
>>>> > > Volker,
>>>> > >
>>>> > > With mpi_f08, you have to declare
>>>> > >
>>>> > > Type(MPI_Comm) :: mpi_comm_global
>>>> > >
>>>> > > (I am afk and not 100% sure of the syntax)
>>>> > >
>>>> > > A simpler option is to
>>>> > >
>>>> > > use mpi
>>>> > >
>>>> > > Cheers,
>>>> > >
>>>> > > Gilles
>>>> > >
>>>> > > Volker Blum >>> wrote:
>>>> > >> Hi Gilles,
>>>> > >>
>>>> > >> Thank you very much for the response!
>>>> > >>
>>>> > >> Unfortunately, I don’t have access to a different system
>>>with the issue right now. As I said, it’s not new; it just keeps
>>>creeping up unexpectedly again on different platforms. What
>>>puzzles me is that I’ve encountered the same problem with low but
>>>reasonable frequency over a period of now over five years.
>>>> > >>

Re: [OMPI users] OpenMPI 1.10.7 and Infiniband

[Russell Dekema]

Are you sure your InfiniBand network is up and running? What kind of
output do you get if you run the command 'ibv_devinfo'?

Sincerely,
Rusty Dekema

On Wed, Jul 26, 2017 at 2:40 PM, Sajesh Singh  wrote:
> OS: Centos 7
>
> Infiniband Packages from OS repos
>
> Mellanox HCA
>
>
>
>
>
> Compiled openmpi 1.10.7 on centos7 with the following config
>
>
>
> ./configure --prefix=/usr/local/software/OpenMPI/openmpi-1.10.7
> --with-tm=/opt/pbs --with-verbs
>
>
>
> Snippet from config.log seems to indicate that the infiniband header files
> were located
>
>
>
> btl_openib_CPPFLAGS=' -I/usr/include/infiniband'
>
> common_verbs_CPPFLAGS=' -I/usr/include/infiniband'
>
> oshmem_verbs_CPPFLAGS=' -I/usr/include/infiniband'
>
>
>
> Everthing seems to have compiled correctly, but when I try to run any
> program using mpirun I am receiving the following error:
>
>
>
> mpirun -np 8 ./a.out
>
> --
>
> [[18431,1],2]: A high-performance Open MPI point-to-point messaging module
>
> was unable to find any relevant network interfaces:
>
>
>
> Module: OpenFabrics (openib)
>
>   Host: host-name
>
>
>
> Another transport will be used instead, although this may result in
>
> lower performance.
>
> --
>
> [:13959] 7 more processes have sent help message
> help-mpi-btl-base.txt / btl:no-nics
>
> [:13959] Set MCA parameter "orte_base_help_aggre
> gate" to 0 to see all help / error messages
>
>
>
>
>
> I am unsure as to where to go from here. Any help would be appreciated as to
> how to troubleshoot this issue.
>
>
>
> Thank you,
>
>
>
> Sajesh
>
>
> ___
> users mailing list
> users@lists.open-mpi.org
> https://rfd.newmexicoconsortium.org/mailman/listinfo/users
___
users mailing list
users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

Re: [OMPI users] MPI_IN_PLACE

[Volker Blum]

Thanks! That’s great. Sounds like the exact combination I have here.

Thanks also to George. Sorry that the test did not trigger on a more standard 
platform - that would have simplified things.

Best wishes
Volker

> On Jul 27, 2017, at 3:56 AM, Gilles Gouaillardet  wrote:
> 
> Folks,
> 
> 
> I am able to reproduce the issue on OS X (Sierra) with stock gcc (aka clang) 
> and ifort 17.0.4
> 
> 
> i will investigate this from now
> 
> 
> Cheers,
> 
> Gilles
> 
> On 7/27/2017 9:28 AM, George Bosilca wrote:
>> Volker,
>> 
>> Unfortunately, I can't replicate with icc. I tried on a x86_64 box with 
>> Intel compiler chain 17.0.4 20170411 to no avail. I also tested the 
>> 3.0.0-rc1 tarball and the current master, and you test completes without 
>> errors on all cases.
>> 
>> Once you figure out an environment where you can consistently replicate the 
>> issue, I would suggest to attach to the processes and:
>> - make sure the MPI_IN_PLACE as seen through the Fortran layer matches what 
>> the C layer expects
>> - what is the collective algorithm used by Open MPI
>> 
>> I have a "Fortran 101" level question. When you pass an array a(:) as 
>> argument, what exactly gets passed via the Fortran interface to the 
>> corresponding C function ?
>> 
>>  George.
>> 
>> On Wed, Jul 26, 2017 at 1:55 PM, Volker Blum > > wrote:
>> 
>>Thanks! Yes, trying with Intel 2017 would be very nice.
>> 
>>> On Jul 26, 2017, at 6:12 PM, George Bosilca >> wrote:
>>>
>>> No, I don't have (or used where they were available) the Intel
>>compiler. I used clang and gfortran. I can try on a Linux box with
>>the Intel 2017 compilers.
>>>
>>>   George.
>>>
>>>
>>>
>>> On Wed, Jul 26, 2017 at 11:59 AM, Volker Blum
>>> wrote:
>>> Did you use Intel Fortran 2017 as well?
>>>
>>> (I’m asking because I did see the same issue with a combination
>>of an earlier Intel Fortran 2017 version and OpenMPI on an
>>Intel/Infiniband Linux HPC machine … but not Intel Fortran 2016 on
>>the same machine. Perhaps I can revive my access to that
>>combination somehow.)
>>>
>>> Best wishes
>>> Volker
>>>
>>> > On Jul 26, 2017, at 5:55 PM, George Bosilca
>>> wrote:
>>> >
>>> > I thought that maybe the underlying allreduce algorithm fails
>>to support MPI_IN_PLACE correctly, but I can't replicate on any
>>machine (including OSX) with any number of processes.
>>> >
>>> >   George.
>>> >
>>> >
>>> >
>>> > On Wed, Jul 26, 2017 at 10:59 AM, Volker Blum
>>> wrote:
>>> > Thanks!
>>> >
>>> > I tried ‘use mpi’, which compiles fine.
>>> >
>>> > Same result as with ‘include mpif.h', in that the output is
>>> >
>>> >  * MPI_IN_PLACE does not appear to work as intended.
>>> >  * Checking whether MPI_ALLREDUCE works at all.
>>> >  * Without MPI_IN_PLACE, MPI_ALLREDUCE appears to work.
>>> >
>>> > Hm. Any other thoughts?
>>> >
>>> > Thanks again!
>>> > Best wishes
>>> > Volker
>>> >
>>> > > On Jul 26, 2017, at 4:06 PM, Gilles Gouaillardet
>>>> wrote:
>>> > >
>>> > > Volker,
>>> > >
>>> > > With mpi_f08, you have to declare
>>> > >
>>> > > Type(MPI_Comm) :: mpi_comm_global
>>> > >
>>> > > (I am afk and not 100% sure of the syntax)
>>> > >
>>> > > A simpler option is to
>>> > >
>>> > > use mpi
>>> > >
>>> > > Cheers,
>>> > >
>>> > > Gilles
>>> > >
>>> > > Volker Blum >> wrote:
>>> > >> Hi Gilles,
>>> > >>
>>> > >> Thank you very much for the response!
>>> > >>
>>> > >> Unfortunately, I don’t have access to a different system
>>with the issue right now. As I said, it’s not new; it just keeps
>>creeping up unexpectedly again on different platforms. What
>>puzzles me is that I’ve encountered the same problem with low but
>>reasonable frequency over a period of now over five years.
>>> > >>
>>> > >> We can’t require F’08 in our application, unfortunately,
>>since this standard is too new. Since we maintain a large
>>application that has to run on a broad range of platforms, Fortran
>>2008 would not work for many of our users. In a few years, this
>>will be different, but not yet.
>>> > >>
>>> > >> On gfortran: In our own tests, unfortunately, Intel Fortran
>>consistently produced much faster executable code in the past. The
>>latter observation may also change someday, but for us, the
>>performance difference was an 

Re: [OMPI users] MPI_IN_PLACE

[Jeff Hammond]

Does this happen with ifort but not other Fortran compilers? If so, write
me off-list if there's a need to report a compiler issue.

Jeff

On Wed, Jul 26, 2017 at 6:59 PM Gilles Gouaillardet 
wrote:

> Folks,
>
>
> I am able to reproduce the issue on OS X (Sierra) with stock gcc (aka
> clang) and ifort 17.0.4
>
>
> i will investigate this from now
>
>
> Cheers,
>
> Gilles
>
> On 7/27/2017 9:28 AM, George Bosilca wrote:
> > Volker,
> >
> > Unfortunately, I can't replicate with icc. I tried on a x86_64 box
> > with Intel compiler chain 17.0.4 20170411 to no avail. I also tested
> > the 3.0.0-rc1 tarball and the current master, and you test completes
> > without errors on all cases.
> >
> > Once you figure out an environment where you can consistently
> > replicate the issue, I would suggest to attach to the processes and:
> > - make sure the MPI_IN_PLACE as seen through the Fortran layer matches
> > what the C layer expects
> > - what is the collective algorithm used by Open MPI
> >
> > I have a "Fortran 101" level question. When you pass an array a(:) as
> > argument, what exactly gets passed via the Fortran interface to the
> > corresponding C function ?
> >
> >   George.
> >
> > On Wed, Jul 26, 2017 at 1:55 PM, Volker Blum  > > wrote:
> >
> > Thanks! Yes, trying with Intel 2017 would be very nice.
> >
> > > On Jul 26, 2017, at 6:12 PM, George Bosilca  > > wrote:
> > >
> > > No, I don't have (or used where they were available) the Intel
> > compiler. I used clang and gfortran. I can try on a Linux box with
> > the Intel 2017 compilers.
> > >
> > >   George.
> > >
> > >
> > >
> > > On Wed, Jul 26, 2017 at 11:59 AM, Volker Blum
> > > wrote:
> > > Did you use Intel Fortran 2017 as well?
> > >
> > > (I’m asking because I did see the same issue with a combination
> > of an earlier Intel Fortran 2017 version and OpenMPI on an
> > Intel/Infiniband Linux HPC machine … but not Intel Fortran 2016 on
> > the same machine. Perhaps I can revive my access to that
> > combination somehow.)
> > >
> > > Best wishes
> > > Volker
> > >
> > > > On Jul 26, 2017, at 5:55 PM, George Bosilca
> > > wrote:
> > > >
> > > > I thought that maybe the underlying allreduce algorithm fails
> > to support MPI_IN_PLACE correctly, but I can't replicate on any
> > machine (including OSX) with any number of processes.
> > > >
> > > >   George.
> > > >
> > > >
> > > >
> > > > On Wed, Jul 26, 2017 at 10:59 AM, Volker Blum
> > > wrote:
> > > > Thanks!
> > > >
> > > > I tried ‘use mpi’, which compiles fine.
> > > >
> > > > Same result as with ‘include mpif.h', in that the output is
> > > >
> > > >  * MPI_IN_PLACE does not appear to work as intended.
> > > >  * Checking whether MPI_ALLREDUCE works at all.
> > > >  * Without MPI_IN_PLACE, MPI_ALLREDUCE appears to work.
> > > >
> > > > Hm. Any other thoughts?
> > > >
> > > > Thanks again!
> > > > Best wishes
> > > > Volker
> > > >
> > > > > On Jul 26, 2017, at 4:06 PM, Gilles Gouaillardet
> >  > > wrote:
> > > > >
> > > > > Volker,
> > > > >
> > > > > With mpi_f08, you have to declare
> > > > >
> > > > > Type(MPI_Comm) :: mpi_comm_global
> > > > >
> > > > > (I am afk and not 100% sure of the syntax)
> > > > >
> > > > > A simpler option is to
> > > > >
> > > > > use mpi
> > > > >
> > > > > Cheers,
> > > > >
> > > > > Gilles
> > > > >
> > > > > Volker Blum  > > wrote:
> > > > >> Hi Gilles,
> > > > >>
> > > > >> Thank you very much for the response!
> > > > >>
> > > > >> Unfortunately, I don’t have access to a different system
> > with the issue right now. As I said, it’s not new; it just keeps
> > creeping up unexpectedly again on different platforms. What
> > puzzles me is that I’ve encountered the same problem with low but
> > reasonable frequency over a period of now over five years.
> > > > >>
> > > > >> We can’t require F’08 in our application, unfortunately,
> > since this standard is too new. Since we maintain a large
> > application that has to run on a broad range of platforms, Fortran
> > 2008 would not work for many of our users. In a few years, this
> > will be different, but not yet.
> > > > >>
> > > > >> On gfortran: In our own tests, unfortunately, Intel Fortran
> > consistently produced much faster executable code in the past. The

Re: [OMPI users] MPI_IN_PLACE

[Gilles Gouaillardet]

Folks,


I am able to reproduce the issue on OS X (Sierra) with stock gcc (aka 
clang) and ifort 17.0.4



i will investigate this from now


Cheers,

Gilles

On 7/27/2017 9:28 AM, George Bosilca wrote:

Volker,

Unfortunately, I can't replicate with icc. I tried on a x86_64 box 
with Intel compiler chain 17.0.4 20170411 to no avail. I also tested 
the 3.0.0-rc1 tarball and the current master, and you test completes 
without errors on all cases.


Once you figure out an environment where you can consistently 
replicate the issue, I would suggest to attach to the processes and:
- make sure the MPI_IN_PLACE as seen through the Fortran layer matches 
what the C layer expects

- what is the collective algorithm used by Open MPI

I have a "Fortran 101" level question. When you pass an array a(:) as 
argument, what exactly gets passed via the Fortran interface to the 
corresponding C function ?


  George.

On Wed, Jul 26, 2017 at 1:55 PM, Volker Blum > wrote:


Thanks! Yes, trying with Intel 2017 would be very nice.

> On Jul 26, 2017, at 6:12 PM, George Bosilca > wrote:
>
> No, I don't have (or used where they were available) the Intel
compiler. I used clang and gfortran. I can try on a Linux box with
the Intel 2017 compilers.
>
>   George.
>
>
>
> On Wed, Jul 26, 2017 at 11:59 AM, Volker Blum
> wrote:
> Did you use Intel Fortran 2017 as well?
>
> (I’m asking because I did see the same issue with a combination
of an earlier Intel Fortran 2017 version and OpenMPI on an
Intel/Infiniband Linux HPC machine … but not Intel Fortran 2016 on
the same machine. Perhaps I can revive my access to that
combination somehow.)
>
> Best wishes
> Volker
>
> > On Jul 26, 2017, at 5:55 PM, George Bosilca
> wrote:
> >
> > I thought that maybe the underlying allreduce algorithm fails
to support MPI_IN_PLACE correctly, but I can't replicate on any
machine (including OSX) with any number of processes.
> >
> >   George.
> >
> >
> >
> > On Wed, Jul 26, 2017 at 10:59 AM, Volker Blum
> wrote:
> > Thanks!
> >
> > I tried ‘use mpi’, which compiles fine.
> >
> > Same result as with ‘include mpif.h', in that the output is
> >
> >  * MPI_IN_PLACE does not appear to work as intended.
> >  * Checking whether MPI_ALLREDUCE works at all.
> >  * Without MPI_IN_PLACE, MPI_ALLREDUCE appears to work.
> >
> > Hm. Any other thoughts?
> >
> > Thanks again!
> > Best wishes
> > Volker
> >
> > > On Jul 26, 2017, at 4:06 PM, Gilles Gouaillardet
> wrote:
> > >
> > > Volker,
> > >
> > > With mpi_f08, you have to declare
> > >
> > > Type(MPI_Comm) :: mpi_comm_global
> > >
> > > (I am afk and not 100% sure of the syntax)
> > >
> > > A simpler option is to
> > >
> > > use mpi
> > >
> > > Cheers,
> > >
> > > Gilles
> > >
> > > Volker Blum > wrote:
> > >> Hi Gilles,
> > >>
> > >> Thank you very much for the response!
> > >>
> > >> Unfortunately, I don’t have access to a different system
with the issue right now. As I said, it’s not new; it just keeps
creeping up unexpectedly again on different platforms. What
puzzles me is that I’ve encountered the same problem with low but
reasonable frequency over a period of now over five years.
> > >>
> > >> We can’t require F’08 in our application, unfortunately,
since this standard is too new. Since we maintain a large
application that has to run on a broad range of platforms, Fortran
2008 would not work for many of our users. In a few years, this
will be different, but not yet.
> > >>
> > >> On gfortran: In our own tests, unfortunately, Intel Fortran
consistently produced much faster executable code in the past. The
latter observation may also change someday, but for us, the
performance difference was an important constraint.
> > >>
> > >> I did suspect mpif.h, too. Not sure how to best test this
hypothesis, however.
> > >>
> > >> Just replacing
> > >>
> > >>> include 'mpif.h'
> > >>> with
> > >>> use mpi_f08
> > >>
> > >> did not work, for me.
> > >>
> > >> This produces a number of compilation errors:
> > >>
> > >> blum:/Users/blum/codes/fhi-aims/openmpi_test> mpif90
check_mpi_in_place_08.f90 -o check_mpi_in_place_08.x
> > >> check_mpi_in_place_08.f90(55): error #6303: The assignment
operation or the binary 

Re: [OMPI users] Questions about integration with resource distribution systems

[r...@open-mpi.org]

Oh no, that's not right. Mpirun launches daemons using qrsh and those daemons 
spawn the app's procs. SGE has no visibility of the app at all

Sent from my iPad

> On Jul 26, 2017, at 7:46 AM, Kulshrestha, Vipul 
>  wrote:
> 
> Thanks Reuti & RHC for your responses.
> 
> My application does not relies on the actual value of m_mem_free and I used 
> this as an example, in open source SGE environment, we use mem_free resource.
> 
> Now, I understand that SGE will allocate requested resources (based on qsub 
> options) and then launch mpirun, which starts "a.out" on allocated resouces 
> using 'qrsh -inherit', so that SGE can keep track of all the launched 
> processes.
> 
> I assume LSF integration works in a similar way.
> 
> Regards,
> Vipul
> 
> 
> -Original Message-
> From: users [mailto:users-boun...@lists.open-mpi.org] On Behalf Of Reuti
> Sent: Wednesday, July 26, 2017 9:25 AM
> To: Open MPI Users 
> Subject: Re: [OMPI users] Questions about integration with resource 
> distribution systems
> 
> 
>> Am 26.07.2017 um 15:09 schrieb r...@open-mpi.org:
>> 
>> mpirun doesn’t get access to that requirement, nor does it need to do so. 
>> SGE will use the requirement when determining the nodes to allocate.
> 
> m_mem_free appears to come from Univa GE and is not part of the open source 
> versions. So I can't comment on this for sure, but it seems to set the memory 
> also in cgroups.
> 
> -- Reuti
> 
> 
>> mpirun just uses the nodes that SGE provides.
>> 
>> What your cmd line does is restrict the entire operation on each node 
>> (daemon + 8 procs) to 40GB of memory. OMPI does not support per-process 
>> restrictions other than binding to cpus.
>> 
>> 
>>> On Jul 26, 2017, at 6:03 AM, Kulshrestha, Vipul 
>>>  wrote:
>>> 
>>> Thanks for a quick response.
>>> 
>>> I will try building OMPI as suggested.
>>> 
>>> On the integration with unsupported distribution systems, we cannot use 
>>> script based approach, because often these machines don’t have ssh 
>>> permission in customer environment. I will explore the path of writing orte 
>>> component. At this stage, I don’t understand the effort for the same.
>>> 
>>> I guess my question 2 was not understood correctly. I used the below 
>>> command as an example for SGE and want to understand the expected behavior 
>>> for such a command. With the below command, I expect to have 8 copies of 
>>> a.out launched with each copy having access to 40GB of memory. Is that 
>>> correct? I am doubtful, because I don’t understand how mpirun gets access 
>>> to information about RAM requirement.
>>> 
>>> qsub –pe orte 8 –b y –V –l m_mem_free=40G –cwd mpirun –np 8 a.out
>>> 
>>> 
>>> Regards,
>>> Vipul
>>> 
>>> 
>>> 
>>> From: users [mailto:users-boun...@lists.open-mpi.org] On Behalf Of 
>>> r...@open-mpi.org
>>> Sent: Tuesday, July 25, 2017 8:16 PM
>>> To: Open MPI Users 
>>> Subject: Re: [OMPI users] Questions about integration with resource 
>>> distribution systems
>>> 
>>> 
>>> On Jul 25, 2017, at 3:48 PM, Kulshrestha, Vipul 
>>>  wrote:
>>> 
>>> I have several questions about integration of openmpi with resource queuing 
>>> systems.
>>> 
>>> 1.
>>> I understand that openmpi supports integration with various resource 
>>> distribution systems such as SGE, LSF, torque etc.
>>> 
>>> I need to build an openmpi application that can interact with variety of 
>>> different resource distribution systems, since different customers have 
>>> different systems. Based on my research, it seems that I need to build a 
>>> different openmpi installation to work, e.g. create an installation of 
>>> opempi with grid and create a different installation of openmpi with LSF. 
>>> Is there a way to build a generic installation of openmpi that can be used 
>>> with more than 1 distribution system by using some generic mechanism?
>>> 
>>> Just to be clear: your application doesn’t depend on the environment in any 
>>> way. Only mpirun does - so if you are distributing an _application_, then 
>>> your question is irrelevant. 
>>> 
>>> If you are distributing OMPI itself, and therefore mpirun, then you can 
>>> build the various components if you first install the headers for that 
>>> environment on your system. It means that you need one machine where all 
>>> those resource managers at least have their headers installed on it. Then 
>>> configure OMPI --with-xxx pointing to each of the RM’s headers so all the 
>>> components get built. When the binary hits your customer’s machine, only 
>>> those components that have active libraries present will execute.
>>> 
>>> 
>>> 2.
>>> For integration with LSF/grid, how would I specify the memory (RAM) 
>>> requirement (or some other parameter) to bsub/qsub, when launching mpirun 
>>> command? Will something like below work to ensure that each of the 8 copies 
>>> of a.out have 40 GB memory reserved 

Re: [OMPI users] MPI_IN_PLACE

[George Bosilca]

Volker,

Unfortunately, I can't replicate with icc. I tried on a x86_64 box with
Intel compiler chain 17.0.4 20170411 to no avail. I also tested the
3.0.0-rc1 tarball and the current master, and you test completes without
errors on all cases.

Once you figure out an environment where you can consistently replicate the
issue, I would suggest to attach to the processes and:
- make sure the MPI_IN_PLACE as seen through the Fortran layer matches what
the C layer expects
- what is the collective algorithm used by Open MPI

I have a "Fortran 101" level question. When you pass an array a(:) as
argument, what exactly gets passed via the Fortran interface to the
corresponding C function ?

  George.

On Wed, Jul 26, 2017 at 1:55 PM, Volker Blum  wrote:

> Thanks! Yes, trying with Intel 2017 would be very nice.
>
> > On Jul 26, 2017, at 6:12 PM, George Bosilca  wrote:
> >
> > No, I don't have (or used where they were available) the Intel compiler.
> I used clang and gfortran. I can try on a Linux box with the Intel 2017
> compilers.
> >
> >   George.
> >
> >
> >
> > On Wed, Jul 26, 2017 at 11:59 AM, Volker Blum 
> wrote:
> > Did you use Intel Fortran 2017 as well?
> >
> > (I’m asking because I did see the same issue with a combination of an
> earlier Intel Fortran 2017 version and OpenMPI on an Intel/Infiniband Linux
> HPC machine … but not Intel Fortran 2016 on the same machine. Perhaps I can
> revive my access to that combination somehow.)
> >
> > Best wishes
> > Volker
> >
> > > On Jul 26, 2017, at 5:55 PM, George Bosilca 
> wrote:
> > >
> > > I thought that maybe the underlying allreduce algorithm fails to
> support MPI_IN_PLACE correctly, but I can't replicate on any machine
> (including OSX) with any number of processes.
> > >
> > >   George.
> > >
> > >
> > >
> > > On Wed, Jul 26, 2017 at 10:59 AM, Volker Blum 
> wrote:
> > > Thanks!
> > >
> > > I tried ‘use mpi’, which compiles fine.
> > >
> > > Same result as with ‘include mpif.h', in that the output is
> > >
> > >  * MPI_IN_PLACE does not appear to work as intended.
> > >  * Checking whether MPI_ALLREDUCE works at all.
> > >  * Without MPI_IN_PLACE, MPI_ALLREDUCE appears to work.
> > >
> > > Hm. Any other thoughts?
> > >
> > > Thanks again!
> > > Best wishes
> > > Volker
> > >
> > > > On Jul 26, 2017, at 4:06 PM, Gilles Gouaillardet <
> gilles.gouaillar...@gmail.com> wrote:
> > > >
> > > > Volker,
> > > >
> > > > With mpi_f08, you have to declare
> > > >
> > > > Type(MPI_Comm) :: mpi_comm_global
> > > >
> > > > (I am afk and not 100% sure of the syntax)
> > > >
> > > > A simpler option is to
> > > >
> > > > use mpi
> > > >
> > > > Cheers,
> > > >
> > > > Gilles
> > > >
> > > > Volker Blum  wrote:
> > > >> Hi Gilles,
> > > >>
> > > >> Thank you very much for the response!
> > > >>
> > > >> Unfortunately, I don’t have access to a different system with the
> issue right now. As I said, it’s not new; it just keeps creeping up
> unexpectedly again on different platforms. What puzzles me is that I’ve
> encountered the same problem with low but reasonable frequency over a
> period of now over five years.
> > > >>
> > > >> We can’t require F’08 in our application, unfortunately, since this
> standard is too new. Since we maintain a large application that has to run
> on a broad range of platforms, Fortran 2008 would not work for many of our
> users. In a few years, this will be different, but not yet.
> > > >>
> > > >> On gfortran: In our own tests, unfortunately, Intel Fortran
> consistently produced much faster executable code in the past. The latter
> observation may also change someday, but for us, the performance difference
> was an important constraint.
> > > >>
> > > >> I did suspect mpif.h, too. Not sure how to best test this
> hypothesis, however.
> > > >>
> > > >> Just replacing
> > > >>
> > > >>> include 'mpif.h'
> > > >>> with
> > > >>> use mpi_f08
> > > >>
> > > >> did not work, for me.
> > > >>
> > > >> This produces a number of compilation errors:
> > > >>
> > > >> blum:/Users/blum/codes/fhi-aims/openmpi_test> mpif90
> check_mpi_in_place_08.f90 -o check_mpi_in_place_08.x
> > > >> check_mpi_in_place_08.f90(55): error #6303: The assignment
> operation or the binary expression operation is invalid for the data types
> of the two operands.   [MPI_COMM_WORLD]
> > > >>   mpi_comm_global = MPI_COMM_WORLD
> > > >> --^
> > > >> check_mpi_in_place_08.f90(57): error #6285: There is no matching
> specific subroutine for this generic subroutine call.   [MPI_COMM_SIZE]
> > > >>   call MPI_COMM_SIZE(mpi_comm_global, n_tasks, mpierr)
> > > >> -^
> > > >> check_mpi_in_place_08.f90(58): error #6285: There is no matching
> specific subroutine for this generic subroutine call.   [MPI_COMM_RANK]
> > > >>   call MPI_COMM_RANK(mpi_comm_global, myid, mpierr)
> > > >> -^
> > > >> check_mpi_in_place_08.f90(75): 

[OMPI users] OpenMPI 1.10.7 and Infiniband

[Sajesh Singh]

OS: Centos 7
Infiniband Packages from OS repos
Mellanox HCA


Compiled openmpi 1.10.7 on centos7 with the following config

./configure --prefix=/usr/local/software/OpenMPI/openmpi-1.10.7 
--with-tm=/opt/pbs --with-verbs

Snippet from config.log seems to indicate that the infiniband header files were 
located

btl_openib_CPPFLAGS=' -I/usr/include/infiniband'
common_verbs_CPPFLAGS=' -I/usr/include/infiniband'
oshmem_verbs_CPPFLAGS=' -I/usr/include/infiniband'

Everthing seems to have compiled correctly, but when I try to run any program 
using mpirun I am receiving the following error:

mpirun -np 8 ./a.out
--
[[18431,1],2]: A high-performance Open MPI point-to-point messaging module
was unable to find any relevant network interfaces:

Module: OpenFabrics (openib)
  Host: host-name

Another transport will be used instead, although this may result in
lower performance.
--
[:13959] 7 more processes have sent help message
   
help-mpi-btl-base.txt / btl:no-nics
[:13959] Set MCA parameter "orte_base_help_aggre
  gate" to 0 to 
see all help / error messages


I am unsure as to where to go from here. Any help would be appreciated as to 
how to troubleshoot this issue.

Thank you,

Sajesh
___
users mailing list
users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

Re: [OMPI users] MPI_IN_PLACE

[Volker Blum]

Thanks! Yes, trying with Intel 2017 would be very nice. 

> On Jul 26, 2017, at 6:12 PM, George Bosilca  wrote:
> 
> No, I don't have (or used where they were available) the Intel compiler. I 
> used clang and gfortran. I can try on a Linux box with the Intel 2017 
> compilers.
> 
>   George.
> 
> 
> 
> On Wed, Jul 26, 2017 at 11:59 AM, Volker Blum  wrote:
> Did you use Intel Fortran 2017 as well?
> 
> (I’m asking because I did see the same issue with a combination of an earlier 
> Intel Fortran 2017 version and OpenMPI on an Intel/Infiniband Linux HPC 
> machine … but not Intel Fortran 2016 on the same machine. Perhaps I can 
> revive my access to that combination somehow.)
> 
> Best wishes
> Volker
> 
> > On Jul 26, 2017, at 5:55 PM, George Bosilca  wrote:
> >
> > I thought that maybe the underlying allreduce algorithm fails to support 
> > MPI_IN_PLACE correctly, but I can't replicate on any machine (including 
> > OSX) with any number of processes.
> >
> >   George.
> >
> >
> >
> > On Wed, Jul 26, 2017 at 10:59 AM, Volker Blum  wrote:
> > Thanks!
> >
> > I tried ‘use mpi’, which compiles fine.
> >
> > Same result as with ‘include mpif.h', in that the output is
> >
> >  * MPI_IN_PLACE does not appear to work as intended.
> >  * Checking whether MPI_ALLREDUCE works at all.
> >  * Without MPI_IN_PLACE, MPI_ALLREDUCE appears to work.
> >
> > Hm. Any other thoughts?
> >
> > Thanks again!
> > Best wishes
> > Volker
> >
> > > On Jul 26, 2017, at 4:06 PM, Gilles Gouaillardet 
> > >  wrote:
> > >
> > > Volker,
> > >
> > > With mpi_f08, you have to declare
> > >
> > > Type(MPI_Comm) :: mpi_comm_global
> > >
> > > (I am afk and not 100% sure of the syntax)
> > >
> > > A simpler option is to
> > >
> > > use mpi
> > >
> > > Cheers,
> > >
> > > Gilles
> > >
> > > Volker Blum  wrote:
> > >> Hi Gilles,
> > >>
> > >> Thank you very much for the response!
> > >>
> > >> Unfortunately, I don’t have access to a different system with the issue 
> > >> right now. As I said, it’s not new; it just keeps creeping up 
> > >> unexpectedly again on different platforms. What puzzles me is that I’ve 
> > >> encountered the same problem with low but reasonable frequency over a 
> > >> period of now over five years.
> > >>
> > >> We can’t require F’08 in our application, unfortunately, since this 
> > >> standard is too new. Since we maintain a large application that has to 
> > >> run on a broad range of platforms, Fortran 2008 would not work for many 
> > >> of our users. In a few years, this will be different, but not yet.
> > >>
> > >> On gfortran: In our own tests, unfortunately, Intel Fortran consistently 
> > >> produced much faster executable code in the past. The latter observation 
> > >> may also change someday, but for us, the performance difference was an 
> > >> important constraint.
> > >>
> > >> I did suspect mpif.h, too. Not sure how to best test this hypothesis, 
> > >> however.
> > >>
> > >> Just replacing
> > >>
> > >>> include 'mpif.h'
> > >>> with
> > >>> use mpi_f08
> > >>
> > >> did not work, for me.
> > >>
> > >> This produces a number of compilation errors:
> > >>
> > >> blum:/Users/blum/codes/fhi-aims/openmpi_test> mpif90 
> > >> check_mpi_in_place_08.f90 -o check_mpi_in_place_08.x
> > >> check_mpi_in_place_08.f90(55): error #6303: The assignment operation or 
> > >> the binary expression operation is invalid for the data types of the two 
> > >> operands.   [MPI_COMM_WORLD]
> > >>   mpi_comm_global = MPI_COMM_WORLD
> > >> --^
> > >> check_mpi_in_place_08.f90(57): error #6285: There is no matching 
> > >> specific subroutine for this generic subroutine call.   [MPI_COMM_SIZE]
> > >>   call MPI_COMM_SIZE(mpi_comm_global, n_tasks, mpierr)
> > >> -^
> > >> check_mpi_in_place_08.f90(58): error #6285: There is no matching 
> > >> specific subroutine for this generic subroutine call.   [MPI_COMM_RANK]
> > >>   call MPI_COMM_RANK(mpi_comm_global, myid, mpierr)
> > >> -^
> > >> check_mpi_in_place_08.f90(75): error #6285: There is no matching 
> > >> specific subroutine for this generic subroutine call.   [MPI_ALLREDUCE]
> > >>   call MPI_ALLREDUCE(MPI_IN_PLACE, &
> > >> -^
> > >> check_mpi_in_place_08.f90(94): error #6285: There is no matching 
> > >> specific subroutine for this generic subroutine call.   [MPI_ALLREDUCE]
> > >>   call MPI_ALLREDUCE(check_success, aux_check_success, 1, MPI_LOGICAL, &
> > >> -^
> > >> check_mpi_in_place_08.f90(119): error #6285: There is no matching 
> > >> specific subroutine for this generic subroutine call.   [MPI_ALLREDUCE]
> > >>  call MPI_ALLREDUCE(test_data(:), &
> > >> ^
> > >> check_mpi_in_place_08.f90(140): error #6285: There is no matching 
> > >> specific subroutine for this generic subroutine call.   [MPI_ALLREDUCE]
> > >>  call 

Re: [OMPI users] MPI_IN_PLACE

[George Bosilca]

No, I don't have (or used where they were available) the Intel compiler. I
used clang and gfortran. I can try on a Linux box with the Intel 2017
compilers.

  George.



On Wed, Jul 26, 2017 at 11:59 AM, Volker Blum  wrote:

> Did you use Intel Fortran 2017 as well?
>
> (I’m asking because I did see the same issue with a combination of an
> earlier Intel Fortran 2017 version and OpenMPI on an Intel/Infiniband Linux
> HPC machine … but not Intel Fortran 2016 on the same machine. Perhaps I can
> revive my access to that combination somehow.)
>
> Best wishes
> Volker
>
> > On Jul 26, 2017, at 5:55 PM, George Bosilca  wrote:
> >
> > I thought that maybe the underlying allreduce algorithm fails to support
> MPI_IN_PLACE correctly, but I can't replicate on any machine (including
> OSX) with any number of processes.
> >
> >   George.
> >
> >
> >
> > On Wed, Jul 26, 2017 at 10:59 AM, Volker Blum 
> wrote:
> > Thanks!
> >
> > I tried ‘use mpi’, which compiles fine.
> >
> > Same result as with ‘include mpif.h', in that the output is
> >
> >  * MPI_IN_PLACE does not appear to work as intended.
> >  * Checking whether MPI_ALLREDUCE works at all.
> >  * Without MPI_IN_PLACE, MPI_ALLREDUCE appears to work.
> >
> > Hm. Any other thoughts?
> >
> > Thanks again!
> > Best wishes
> > Volker
> >
> > > On Jul 26, 2017, at 4:06 PM, Gilles Gouaillardet <
> gilles.gouaillar...@gmail.com> wrote:
> > >
> > > Volker,
> > >
> > > With mpi_f08, you have to declare
> > >
> > > Type(MPI_Comm) :: mpi_comm_global
> > >
> > > (I am afk and not 100% sure of the syntax)
> > >
> > > A simpler option is to
> > >
> > > use mpi
> > >
> > > Cheers,
> > >
> > > Gilles
> > >
> > > Volker Blum  wrote:
> > >> Hi Gilles,
> > >>
> > >> Thank you very much for the response!
> > >>
> > >> Unfortunately, I don’t have access to a different system with the
> issue right now. As I said, it’s not new; it just keeps creeping up
> unexpectedly again on different platforms. What puzzles me is that I’ve
> encountered the same problem with low but reasonable frequency over a
> period of now over five years.
> > >>
> > >> We can’t require F’08 in our application, unfortunately, since this
> standard is too new. Since we maintain a large application that has to run
> on a broad range of platforms, Fortran 2008 would not work for many of our
> users. In a few years, this will be different, but not yet.
> > >>
> > >> On gfortran: In our own tests, unfortunately, Intel Fortran
> consistently produced much faster executable code in the past. The latter
> observation may also change someday, but for us, the performance difference
> was an important constraint.
> > >>
> > >> I did suspect mpif.h, too. Not sure how to best test this hypothesis,
> however.
> > >>
> > >> Just replacing
> > >>
> > >>> include 'mpif.h'
> > >>> with
> > >>> use mpi_f08
> > >>
> > >> did not work, for me.
> > >>
> > >> This produces a number of compilation errors:
> > >>
> > >> blum:/Users/blum/codes/fhi-aims/openmpi_test> mpif90
> check_mpi_in_place_08.f90 -o check_mpi_in_place_08.x
> > >> check_mpi_in_place_08.f90(55): error #6303: The assignment operation
> or the binary expression operation is invalid for the data types of the two
> operands.   [MPI_COMM_WORLD]
> > >>   mpi_comm_global = MPI_COMM_WORLD
> > >> --^
> > >> check_mpi_in_place_08.f90(57): error #6285: There is no matching
> specific subroutine for this generic subroutine call.   [MPI_COMM_SIZE]
> > >>   call MPI_COMM_SIZE(mpi_comm_global, n_tasks, mpierr)
> > >> -^
> > >> check_mpi_in_place_08.f90(58): error #6285: There is no matching
> specific subroutine for this generic subroutine call.   [MPI_COMM_RANK]
> > >>   call MPI_COMM_RANK(mpi_comm_global, myid, mpierr)
> > >> -^
> > >> check_mpi_in_place_08.f90(75): error #6285: There is no matching
> specific subroutine for this generic subroutine call.   [MPI_ALLREDUCE]
> > >>   call MPI_ALLREDUCE(MPI_IN_PLACE, &
> > >> -^
> > >> check_mpi_in_place_08.f90(94): error #6285: There is no matching
> specific subroutine for this generic subroutine call.   [MPI_ALLREDUCE]
> > >>   call MPI_ALLREDUCE(check_success, aux_check_success, 1,
> MPI_LOGICAL, &
> > >> -^
> > >> check_mpi_in_place_08.f90(119): error #6285: There is no matching
> specific subroutine for this generic subroutine call.   [MPI_ALLREDUCE]
> > >>  call MPI_ALLREDUCE(test_data(:), &
> > >> ^
> > >> check_mpi_in_place_08.f90(140): error #6285: There is no matching
> specific subroutine for this generic subroutine call.   [MPI_ALLREDUCE]
> > >>  call MPI_ALLREDUCE(check_conventional_mpi, aux_check_success,
> 1, MPI_LOGICAL, &
> > >> ^
> > >> compilation aborted for check_mpi_in_place_08.f90 (code 1)
> > >>
> > >> This is an interesting result, however … what might I be missing?
> Another use statement?
> > >>
> > >> Best wishes
> > >> Volker
> > >>
> > 

Re: [OMPI users] MPI_IN_PLACE

[Volker Blum]

Did you use Intel Fortran 2017 as well?

(I’m asking because I did see the same issue with a combination of an earlier 
Intel Fortran 2017 version and OpenMPI on an Intel/Infiniband Linux HPC machine 
… but not Intel Fortran 2016 on the same machine. Perhaps I can revive my 
access to that combination somehow.)

Best wishes
Volker

> On Jul 26, 2017, at 5:55 PM, George Bosilca  wrote:
> 
> I thought that maybe the underlying allreduce algorithm fails to support 
> MPI_IN_PLACE correctly, but I can't replicate on any machine (including OSX) 
> with any number of processes.
> 
>   George.
> 
> 
> 
> On Wed, Jul 26, 2017 at 10:59 AM, Volker Blum  wrote:
> Thanks!
> 
> I tried ‘use mpi’, which compiles fine.
> 
> Same result as with ‘include mpif.h', in that the output is
> 
>  * MPI_IN_PLACE does not appear to work as intended.
>  * Checking whether MPI_ALLREDUCE works at all.
>  * Without MPI_IN_PLACE, MPI_ALLREDUCE appears to work.
> 
> Hm. Any other thoughts?
> 
> Thanks again!
> Best wishes
> Volker
> 
> > On Jul 26, 2017, at 4:06 PM, Gilles Gouaillardet 
> >  wrote:
> >
> > Volker,
> >
> > With mpi_f08, you have to declare
> >
> > Type(MPI_Comm) :: mpi_comm_global
> >
> > (I am afk and not 100% sure of the syntax)
> >
> > A simpler option is to
> >
> > use mpi
> >
> > Cheers,
> >
> > Gilles
> >
> > Volker Blum  wrote:
> >> Hi Gilles,
> >>
> >> Thank you very much for the response!
> >>
> >> Unfortunately, I don’t have access to a different system with the issue 
> >> right now. As I said, it’s not new; it just keeps creeping up unexpectedly 
> >> again on different platforms. What puzzles me is that I’ve encountered the 
> >> same problem with low but reasonable frequency over a period of now over 
> >> five years.
> >>
> >> We can’t require F’08 in our application, unfortunately, since this 
> >> standard is too new. Since we maintain a large application that has to run 
> >> on a broad range of platforms, Fortran 2008 would not work for many of our 
> >> users. In a few years, this will be different, but not yet.
> >>
> >> On gfortran: In our own tests, unfortunately, Intel Fortran consistently 
> >> produced much faster executable code in the past. The latter observation 
> >> may also change someday, but for us, the performance difference was an 
> >> important constraint.
> >>
> >> I did suspect mpif.h, too. Not sure how to best test this hypothesis, 
> >> however.
> >>
> >> Just replacing
> >>
> >>> include 'mpif.h'
> >>> with
> >>> use mpi_f08
> >>
> >> did not work, for me.
> >>
> >> This produces a number of compilation errors:
> >>
> >> blum:/Users/blum/codes/fhi-aims/openmpi_test> mpif90 
> >> check_mpi_in_place_08.f90 -o check_mpi_in_place_08.x
> >> check_mpi_in_place_08.f90(55): error #6303: The assignment operation or 
> >> the binary expression operation is invalid for the data types of the two 
> >> operands.   [MPI_COMM_WORLD]
> >>   mpi_comm_global = MPI_COMM_WORLD
> >> --^
> >> check_mpi_in_place_08.f90(57): error #6285: There is no matching specific 
> >> subroutine for this generic subroutine call.   [MPI_COMM_SIZE]
> >>   call MPI_COMM_SIZE(mpi_comm_global, n_tasks, mpierr)
> >> -^
> >> check_mpi_in_place_08.f90(58): error #6285: There is no matching specific 
> >> subroutine for this generic subroutine call.   [MPI_COMM_RANK]
> >>   call MPI_COMM_RANK(mpi_comm_global, myid, mpierr)
> >> -^
> >> check_mpi_in_place_08.f90(75): error #6285: There is no matching specific 
> >> subroutine for this generic subroutine call.   [MPI_ALLREDUCE]
> >>   call MPI_ALLREDUCE(MPI_IN_PLACE, &
> >> -^
> >> check_mpi_in_place_08.f90(94): error #6285: There is no matching specific 
> >> subroutine for this generic subroutine call.   [MPI_ALLREDUCE]
> >>   call MPI_ALLREDUCE(check_success, aux_check_success, 1, MPI_LOGICAL, &
> >> -^
> >> check_mpi_in_place_08.f90(119): error #6285: There is no matching specific 
> >> subroutine for this generic subroutine call.   [MPI_ALLREDUCE]
> >>  call MPI_ALLREDUCE(test_data(:), &
> >> ^
> >> check_mpi_in_place_08.f90(140): error #6285: There is no matching specific 
> >> subroutine for this generic subroutine call.   [MPI_ALLREDUCE]
> >>  call MPI_ALLREDUCE(check_conventional_mpi, aux_check_success, 1, 
> >> MPI_LOGICAL, &
> >> ^
> >> compilation aborted for check_mpi_in_place_08.f90 (code 1)
> >>
> >> This is an interesting result, however … what might I be missing? Another 
> >> use statement?
> >>
> >> Best wishes
> >> Volker
> >>
> >>> On Jul 26, 2017, at 2:53 PM, Gilles Gouaillardet 
> >>>  wrote:
> >>>
> >>> Volker,
> >>>
> >>> thanks, i will have a look at it
> >>>
> >>> meanwhile, if you can reproduce this issue on a more mainstream
> >>> platform (e.g. linux + gfortran) please let me know.
> >>>
> >>> since you are using ifort, Open MPI was 

Re: [OMPI users] MPI_IN_PLACE

[Volker Blum]

Thanks!

I tried ‘use mpi’, which compiles fine.

Same result as with ‘include mpif.h', in that the output is

 * MPI_IN_PLACE does not appear to work as intended.
 * Checking whether MPI_ALLREDUCE works at all.
 * Without MPI_IN_PLACE, MPI_ALLREDUCE appears to work.

Hm. Any other thoughts?

Thanks again!
Best wishes
Volker

> On Jul 26, 2017, at 4:06 PM, Gilles Gouaillardet 
>  wrote:
> 
> Volker,
> 
> With mpi_f08, you have to declare
> 
> Type(MPI_Comm) :: mpi_comm_global
> 
> (I am afk and not 100% sure of the syntax)
> 
> A simpler option is to
> 
> use mpi
> 
> Cheers,
> 
> Gilles
> 
> Volker Blum  wrote:
>> Hi Gilles,
>> 
>> Thank you very much for the response!
>> 
>> Unfortunately, I don’t have access to a different system with the issue 
>> right now. As I said, it’s not new; it just keeps creeping up unexpectedly 
>> again on different platforms. What puzzles me is that I’ve encountered the 
>> same problem with low but reasonable frequency over a period of now over 
>> five years.
>> 
>> We can’t require F’08 in our application, unfortunately, since this standard 
>> is too new. Since we maintain a large application that has to run on a broad 
>> range of platforms, Fortran 2008 would not work for many of our users. In a 
>> few years, this will be different, but not yet.
>> 
>> On gfortran: In our own tests, unfortunately, Intel Fortran consistently 
>> produced much faster executable code in the past. The latter observation may 
>> also change someday, but for us, the performance difference was an important 
>> constraint.
>> 
>> I did suspect mpif.h, too. Not sure how to best test this hypothesis, 
>> however. 
>> 
>> Just replacing 
>> 
>>> include 'mpif.h'
>>> with
>>> use mpi_f08
>> 
>> did not work, for me. 
>> 
>> This produces a number of compilation errors:
>> 
>> blum:/Users/blum/codes/fhi-aims/openmpi_test> mpif90 
>> check_mpi_in_place_08.f90 -o check_mpi_in_place_08.x
>> check_mpi_in_place_08.f90(55): error #6303: The assignment operation or the 
>> binary expression operation is invalid for the data types of the two 
>> operands.   [MPI_COMM_WORLD]
>>   mpi_comm_global = MPI_COMM_WORLD
>> --^
>> check_mpi_in_place_08.f90(57): error #6285: There is no matching specific 
>> subroutine for this generic subroutine call.   [MPI_COMM_SIZE]
>>   call MPI_COMM_SIZE(mpi_comm_global, n_tasks, mpierr)
>> -^
>> check_mpi_in_place_08.f90(58): error #6285: There is no matching specific 
>> subroutine for this generic subroutine call.   [MPI_COMM_RANK]
>>   call MPI_COMM_RANK(mpi_comm_global, myid, mpierr)
>> -^
>> check_mpi_in_place_08.f90(75): error #6285: There is no matching specific 
>> subroutine for this generic subroutine call.   [MPI_ALLREDUCE]
>>   call MPI_ALLREDUCE(MPI_IN_PLACE, &
>> -^
>> check_mpi_in_place_08.f90(94): error #6285: There is no matching specific 
>> subroutine for this generic subroutine call.   [MPI_ALLREDUCE]
>>   call MPI_ALLREDUCE(check_success, aux_check_success, 1, MPI_LOGICAL, &
>> -^
>> check_mpi_in_place_08.f90(119): error #6285: There is no matching specific 
>> subroutine for this generic subroutine call.   [MPI_ALLREDUCE]
>>  call MPI_ALLREDUCE(test_data(:), &
>> ^
>> check_mpi_in_place_08.f90(140): error #6285: There is no matching specific 
>> subroutine for this generic subroutine call.   [MPI_ALLREDUCE]
>>  call MPI_ALLREDUCE(check_conventional_mpi, aux_check_success, 1, 
>> MPI_LOGICAL, &
>> ^
>> compilation aborted for check_mpi_in_place_08.f90 (code 1)
>> 
>> This is an interesting result, however … what might I be missing? Another 
>> use statement?
>> 
>> Best wishes
>> Volker
>> 
>>> On Jul 26, 2017, at 2:53 PM, Gilles Gouaillardet 
>>>  wrote:
>>> 
>>> Volker,
>>> 
>>> thanks, i will have a look at it
>>> 
>>> meanwhile, if you can reproduce this issue on a more mainstream
>>> platform (e.g. linux + gfortran) please let me know.
>>> 
>>> since you are using ifort, Open MPI was built with Fortran 2008
>>> bindings, so you can replace
>>> include 'mpif.h'
>>> with
>>> use mpi_f08
>>> and who knows, that might solve your issue
>>> 
>>> 
>>> Cheers,
>>> 
>>> Gilles
>>> 
>>> On Wed, Jul 26, 2017 at 5:22 PM, Volker Blum  wrote:
 Dear Gilles,
 
 Thank you very much for the fast answer.
 
 Darn. I feared it might not occur on all platforms, since my former Macbook
 (with an older OpenMPI version) no longer exhibited the problem, a 
 different
 Linux/Intel Machine did last December, etc.
 
 On this specific machine, the configure line is
 
 ./configure CC=gcc FC=ifort F77=ifort
 
 ifort version 17.0.4
 
 blum:/Users/blum/software/openmpi-3.0.0rc1> gcc -v
 Configured with: --prefix=/Applications/Xcode.app/Contents/Developer/usr
 --with-gxx-include-dir=/usr/include/c++/4.2.1
 Apple LLVM 

Re: [OMPI users] MPI_IN_PLACE

[Gilles Gouaillardet]

Volker,

With mpi_f08, you have to declare

Type(MPI_Comm) :: mpi_comm_global

(I am afk and not 100% sure of the syntax)

A simpler option is to

use mpi

Cheers,

Gilles

Volker Blum  wrote:
>Hi Gilles,
>
>Thank you very much for the response!
>
>Unfortunately, I don’t have access to a different system with the issue right 
>now. As I said, it’s not new; it just keeps creeping up unexpectedly again on 
>different platforms. What puzzles me is that I’ve encountered the same problem 
>with low but reasonable frequency over a period of now over five years.
>
>We can’t require F’08 in our application, unfortunately, since this standard 
>is too new. Since we maintain a large application that has to run on a broad 
>range of platforms, Fortran 2008 would not work for many of our users. In a 
>few years, this will be different, but not yet.
>
>On gfortran: In our own tests, unfortunately, Intel Fortran consistently 
>produced much faster executable code in the past. The latter observation may 
>also change someday, but for us, the performance difference was an important 
>constraint.
>
>I did suspect mpif.h, too. Not sure how to best test this hypothesis, however. 
>
>Just replacing 
>
>> include 'mpif.h'
>> with
>> use mpi_f08
>
>did not work, for me. 
>
>This produces a number of compilation errors:
>
>blum:/Users/blum/codes/fhi-aims/openmpi_test> mpif90 check_mpi_in_place_08.f90 
>-o check_mpi_in_place_08.x
>check_mpi_in_place_08.f90(55): error #6303: The assignment operation or the 
>binary expression operation is invalid for the data types of the two operands. 
>  [MPI_COMM_WORLD]
>mpi_comm_global = MPI_COMM_WORLD
>--^
>check_mpi_in_place_08.f90(57): error #6285: There is no matching specific 
>subroutine for this generic subroutine call.   [MPI_COMM_SIZE]
>call MPI_COMM_SIZE(mpi_comm_global, n_tasks, mpierr)
>-^
>check_mpi_in_place_08.f90(58): error #6285: There is no matching specific 
>subroutine for this generic subroutine call.   [MPI_COMM_RANK]
>call MPI_COMM_RANK(mpi_comm_global, myid, mpierr)
>-^
>check_mpi_in_place_08.f90(75): error #6285: There is no matching specific 
>subroutine for this generic subroutine call.   [MPI_ALLREDUCE]
>call MPI_ALLREDUCE(MPI_IN_PLACE, &
>-^
>check_mpi_in_place_08.f90(94): error #6285: There is no matching specific 
>subroutine for this generic subroutine call.   [MPI_ALLREDUCE]
>call MPI_ALLREDUCE(check_success, aux_check_success, 1, MPI_LOGICAL, &
>-^
>check_mpi_in_place_08.f90(119): error #6285: There is no matching specific 
>subroutine for this generic subroutine call.   [MPI_ALLREDUCE]
>   call MPI_ALLREDUCE(test_data(:), &
>^
>check_mpi_in_place_08.f90(140): error #6285: There is no matching specific 
>subroutine for this generic subroutine call.   [MPI_ALLREDUCE]
>   call MPI_ALLREDUCE(check_conventional_mpi, aux_check_success, 1, 
> MPI_LOGICAL, &
>^
>compilation aborted for check_mpi_in_place_08.f90 (code 1)
>
>This is an interesting result, however … what might I be missing? Another use 
>statement?
>
>Best wishes
>Volker
>
>> On Jul 26, 2017, at 2:53 PM, Gilles Gouaillardet 
>>  wrote:
>> 
>> Volker,
>> 
>> thanks, i will have a look at it
>> 
>> meanwhile, if you can reproduce this issue on a more mainstream
>> platform (e.g. linux + gfortran) please let me know.
>> 
>> since you are using ifort, Open MPI was built with Fortran 2008
>> bindings, so you can replace
>> include 'mpif.h'
>> with
>> use mpi_f08
>> and who knows, that might solve your issue
>> 
>> 
>> Cheers,
>> 
>> Gilles
>> 
>> On Wed, Jul 26, 2017 at 5:22 PM, Volker Blum  wrote:
>>> Dear Gilles,
>>> 
>>> Thank you very much for the fast answer.
>>> 
>>> Darn. I feared it might not occur on all platforms, since my former Macbook
>>> (with an older OpenMPI version) no longer exhibited the problem, a different
>>> Linux/Intel Machine did last December, etc.
>>> 
>>> On this specific machine, the configure line is
>>> 
>>> ./configure CC=gcc FC=ifort F77=ifort
>>> 
>>> ifort version 17.0.4
>>> 
>>> blum:/Users/blum/software/openmpi-3.0.0rc1> gcc -v
>>> Configured with: --prefix=/Applications/Xcode.app/Contents/Developer/usr
>>> --with-gxx-include-dir=/usr/include/c++/4.2.1
>>> Apple LLVM version 8.1.0 (clang-802.0.42)
>>> Target: x86_64-apple-darwin16.6.0
>>> Thread model: posix
>>> InstalledDir:
>>> /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin
>>> 
>>> The full test program is appended.
>>> 
>>> Compilation:
>>> 
>>> mpif90 check_mpi_in_place.f90
>>> 
>>> blum:/Users/blum/codes/fhi-aims/openmpi_test> which mpif90
>>> /usr/local/openmpi-3.0.0rc1/bin/mpif90
>>> 
>>> blum:/Users/blum/codes/fhi-aims/openmpi_test> which mpirun
>>> /usr/local/openmpi-3.0.0rc1/bin/mpirun
>>> 
>>> blum:/Users/blum/codes/fhi-aims/openmpi_test> mpirun -np 2 a.out
>>> * MPI_IN_PLACE does not appear to 

Re: [OMPI users] MPI_IN_PLACE

[Volker Blum]

Hi Gilles,

Thank you very much for the response!

Unfortunately, I don’t have access to a different system with the issue right 
now. As I said, it’s not new; it just keeps creeping up unexpectedly again on 
different platforms. What puzzles me is that I’ve encountered the same problem 
with low but reasonable frequency over a period of now over five years.

We can’t require F’08 in our application, unfortunately, since this standard is 
too new. Since we maintain a large application that has to run on a broad range 
of platforms, Fortran 2008 would not work for many of our users. In a few 
years, this will be different, but not yet.

On gfortran: In our own tests, unfortunately, Intel Fortran consistently 
produced much faster executable code in the past. The latter observation may 
also change someday, but for us, the performance difference was an important 
constraint.

I did suspect mpif.h, too. Not sure how to best test this hypothesis, however. 

Just replacing 

> include 'mpif.h'
> with
> use mpi_f08

did not work, for me. 

This produces a number of compilation errors:

blum:/Users/blum/codes/fhi-aims/openmpi_test> mpif90 check_mpi_in_place_08.f90 
-o check_mpi_in_place_08.x
check_mpi_in_place_08.f90(55): error #6303: The assignment operation or the 
binary expression operation is invalid for the data types of the two operands.  
 [MPI_COMM_WORLD]
mpi_comm_global = MPI_COMM_WORLD
--^
check_mpi_in_place_08.f90(57): error #6285: There is no matching specific 
subroutine for this generic subroutine call.   [MPI_COMM_SIZE]
call MPI_COMM_SIZE(mpi_comm_global, n_tasks, mpierr)
-^
check_mpi_in_place_08.f90(58): error #6285: There is no matching specific 
subroutine for this generic subroutine call.   [MPI_COMM_RANK]
call MPI_COMM_RANK(mpi_comm_global, myid, mpierr)
-^
check_mpi_in_place_08.f90(75): error #6285: There is no matching specific 
subroutine for this generic subroutine call.   [MPI_ALLREDUCE]
call MPI_ALLREDUCE(MPI_IN_PLACE, &
-^
check_mpi_in_place_08.f90(94): error #6285: There is no matching specific 
subroutine for this generic subroutine call.   [MPI_ALLREDUCE]
call MPI_ALLREDUCE(check_success, aux_check_success, 1, MPI_LOGICAL, &
-^
check_mpi_in_place_08.f90(119): error #6285: There is no matching specific 
subroutine for this generic subroutine call.   [MPI_ALLREDUCE]
   call MPI_ALLREDUCE(test_data(:), &
^
check_mpi_in_place_08.f90(140): error #6285: There is no matching specific 
subroutine for this generic subroutine call.   [MPI_ALLREDUCE]
   call MPI_ALLREDUCE(check_conventional_mpi, aux_check_success, 1, 
MPI_LOGICAL, &
^
compilation aborted for check_mpi_in_place_08.f90 (code 1)

This is an interesting result, however … what might I be missing? Another use 
statement?

Best wishes
Volker

> On Jul 26, 2017, at 2:53 PM, Gilles Gouaillardet 
>  wrote:
> 
> Volker,
> 
> thanks, i will have a look at it
> 
> meanwhile, if you can reproduce this issue on a more mainstream
> platform (e.g. linux + gfortran) please let me know.
> 
> since you are using ifort, Open MPI was built with Fortran 2008
> bindings, so you can replace
> include 'mpif.h'
> with
> use mpi_f08
> and who knows, that might solve your issue
> 
> 
> Cheers,
> 
> Gilles
> 
> On Wed, Jul 26, 2017 at 5:22 PM, Volker Blum  wrote:
>> Dear Gilles,
>> 
>> Thank you very much for the fast answer.
>> 
>> Darn. I feared it might not occur on all platforms, since my former Macbook
>> (with an older OpenMPI version) no longer exhibited the problem, a different
>> Linux/Intel Machine did last December, etc.
>> 
>> On this specific machine, the configure line is
>> 
>> ./configure CC=gcc FC=ifort F77=ifort
>> 
>> ifort version 17.0.4
>> 
>> blum:/Users/blum/software/openmpi-3.0.0rc1> gcc -v
>> Configured with: --prefix=/Applications/Xcode.app/Contents/Developer/usr
>> --with-gxx-include-dir=/usr/include/c++/4.2.1
>> Apple LLVM version 8.1.0 (clang-802.0.42)
>> Target: x86_64-apple-darwin16.6.0
>> Thread model: posix
>> InstalledDir:
>> /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin
>> 
>> The full test program is appended.
>> 
>> Compilation:
>> 
>> mpif90 check_mpi_in_place.f90
>> 
>> blum:/Users/blum/codes/fhi-aims/openmpi_test> which mpif90
>> /usr/local/openmpi-3.0.0rc1/bin/mpif90
>> 
>> blum:/Users/blum/codes/fhi-aims/openmpi_test> which mpirun
>> /usr/local/openmpi-3.0.0rc1/bin/mpirun
>> 
>> blum:/Users/blum/codes/fhi-aims/openmpi_test> mpirun -np 2 a.out
>> * MPI_IN_PLACE does not appear to work as intended.
>> * Checking whether MPI_ALLREDUCE works at all.
>> * Without MPI_IN_PLACE, MPI_ALLREDUCE appears to work.
>> 
>> blum:/Users/blum/codes/fhi-aims/openmpi_test> mpirun -np 1 a.out
>> * MPI_IN_PLACE does not appear to work as intended.
>> * Checking whether MPI_ALLREDUCE works at all.
>> * Without MPI_IN_PLACE, MPI_ALLREDUCE 

Re: [OMPI users] Questions about integration with resource distribution systems

[Kulshrestha, Vipul]

Thanks Reuti & RHC for your responses.

My application does not relies on the actual value of m_mem_free and I used 
this as an example, in open source SGE environment, we use mem_free resource.

Now, I understand that SGE will allocate requested resources (based on qsub 
options) and then launch mpirun, which starts "a.out" on allocated resouces 
using 'qrsh -inherit', so that SGE can keep track of all the launched processes.

I assume LSF integration works in a similar way.

Regards,
Vipul


-Original Message-
From: users [mailto:users-boun...@lists.open-mpi.org] On Behalf Of Reuti
Sent: Wednesday, July 26, 2017 9:25 AM
To: Open MPI Users 
Subject: Re: [OMPI users] Questions about integration with resource 
distribution systems


> Am 26.07.2017 um 15:09 schrieb r...@open-mpi.org:
> 
> mpirun doesn’t get access to that requirement, nor does it need to do so. SGE 
> will use the requirement when determining the nodes to allocate.

m_mem_free appears to come from Univa GE and is not part of the open source 
versions. So I can't comment on this for sure, but it seems to set the memory 
also in cgroups.

-- Reuti


> mpirun just uses the nodes that SGE provides.
> 
> What your cmd line does is restrict the entire operation on each node (daemon 
> + 8 procs) to 40GB of memory. OMPI does not support per-process restrictions 
> other than binding to cpus.
> 
> 
>> On Jul 26, 2017, at 6:03 AM, Kulshrestha, Vipul 
>>  wrote:
>> 
>> Thanks for a quick response.
>>  
>> I will try building OMPI as suggested.
>>  
>> On the integration with unsupported distribution systems, we cannot use 
>> script based approach, because often these machines don’t have ssh 
>> permission in customer environment. I will explore the path of writing orte 
>> component. At this stage, I don’t understand the effort for the same.
>>  
>> I guess my question 2 was not understood correctly. I used the below command 
>> as an example for SGE and want to understand the expected behavior for such 
>> a command. With the below command, I expect to have 8 copies of a.out 
>> launched with each copy having access to 40GB of memory. Is that correct? I 
>> am doubtful, because I don’t understand how mpirun gets access to 
>> information about RAM requirement.
>>  
>> qsub –pe orte 8 –b y –V –l m_mem_free=40G –cwd mpirun –np 8 a.out
>>  
>>  
>> Regards,
>> Vipul
>>  
>>  
>>  
>> From: users [mailto:users-boun...@lists.open-mpi.org] On Behalf Of 
>> r...@open-mpi.org
>> Sent: Tuesday, July 25, 2017 8:16 PM
>> To: Open MPI Users 
>> Subject: Re: [OMPI users] Questions about integration with resource 
>> distribution systems
>>  
>>  
>> On Jul 25, 2017, at 3:48 PM, Kulshrestha, Vipul 
>>  wrote:
>>  
>> I have several questions about integration of openmpi with resource queuing 
>> systems.
>>  
>> 1.
>> I understand that openmpi supports integration with various resource 
>> distribution systems such as SGE, LSF, torque etc.
>>  
>> I need to build an openmpi application that can interact with variety of 
>> different resource distribution systems, since different customers have 
>> different systems. Based on my research, it seems that I need to build a 
>> different openmpi installation to work, e.g. create an installation of 
>> opempi with grid and create a different installation of openmpi with LSF. Is 
>> there a way to build a generic installation of openmpi that can be used with 
>> more than 1 distribution system by using some generic mechanism?
>>  
>> Just to be clear: your application doesn’t depend on the environment in any 
>> way. Only mpirun does - so if you are distributing an _application_, then 
>> your question is irrelevant. 
>>  
>> If you are distributing OMPI itself, and therefore mpirun, then you can 
>> build the various components if you first install the headers for that 
>> environment on your system. It means that you need one machine where all 
>> those resource managers at least have their headers installed on it. Then 
>> configure OMPI --with-xxx pointing to each of the RM’s headers so all the 
>> components get built. When the binary hits your customer’s machine, only 
>> those components that have active libraries present will execute.
>>  
>>  
>> 2.
>> For integration with LSF/grid, how would I specify the memory (RAM) 
>> requirement (or some other parameter) to bsub/qsub, when launching mpirun 
>> command? Will something like below work to ensure that each of the 8 copies 
>> of a.out have 40 GB memory reserved for them by grid engine?
>>  
>> qsub –pe orte 8 –b y –V –l m_mem_free=40G –cwd mpirun –np 8 a.out
>>  
>> You’ll have to provide something that is environment dependent, I’m afraid - 
>> there is no standard out there.
>> 
>> 
>>  
>> 3.
>> Some of our customers use custom distribution engine (some 
>> non-industry-standard distribution engine). How can I integrate my openmpi  
>> 

Re: [OMPI users] Questions about integration with resource distribution systems

[Reuti]

> Am 26.07.2017 um 15:09 schrieb r...@open-mpi.org:
> 
> mpirun doesn’t get access to that requirement, nor does it need to do so. SGE 
> will use the requirement when determining the nodes to allocate.

m_mem_free appears to come from Univa GE and is not part of the open source 
versions. So I can't comment on this for sure, but it seems to set the memory 
also in cgroups.

-- Reuti


> mpirun just uses the nodes that SGE provides.
> 
> What your cmd line does is restrict the entire operation on each node (daemon 
> + 8 procs) to 40GB of memory. OMPI does not support per-process restrictions 
> other than binding to cpus.
> 
> 
>> On Jul 26, 2017, at 6:03 AM, Kulshrestha, Vipul 
>>  wrote:
>> 
>> Thanks for a quick response.
>>  
>> I will try building OMPI as suggested.
>>  
>> On the integration with unsupported distribution systems, we cannot use 
>> script based approach, because often these machines don’t have ssh 
>> permission in customer environment. I will explore the path of writing orte 
>> component. At this stage, I don’t understand the effort for the same.
>>  
>> I guess my question 2 was not understood correctly. I used the below command 
>> as an example for SGE and want to understand the expected behavior for such 
>> a command. With the below command, I expect to have 8 copies of a.out 
>> launched with each copy having access to 40GB of memory. Is that correct? I 
>> am doubtful, because I don’t understand how mpirun gets access to 
>> information about RAM requirement.
>>  
>> qsub –pe orte 8 –b y –V –l m_mem_free=40G –cwd mpirun –np 8 a.out
>>  
>>  
>> Regards,
>> Vipul
>>  
>>  
>>  
>> From: users [mailto:users-boun...@lists.open-mpi.org] On Behalf Of 
>> r...@open-mpi.org
>> Sent: Tuesday, July 25, 2017 8:16 PM
>> To: Open MPI Users 
>> Subject: Re: [OMPI users] Questions about integration with resource 
>> distribution systems
>>  
>>  
>> On Jul 25, 2017, at 3:48 PM, Kulshrestha, Vipul 
>>  wrote:
>>  
>> I have several questions about integration of openmpi with resource queuing 
>> systems.
>>  
>> 1.
>> I understand that openmpi supports integration with various resource 
>> distribution systems such as SGE, LSF, torque etc.
>>  
>> I need to build an openmpi application that can interact with variety of 
>> different resource distribution systems, since different customers have 
>> different systems. Based on my research, it seems that I need to build a 
>> different openmpi installation to work, e.g. create an installation of 
>> opempi with grid and create a different installation of openmpi with LSF. Is 
>> there a way to build a generic installation of openmpi that can be used with 
>> more than 1 distribution system by using some generic mechanism?
>>  
>> Just to be clear: your application doesn’t depend on the environment in any 
>> way. Only mpirun does - so if you are distributing an _application_, then 
>> your question is irrelevant. 
>>  
>> If you are distributing OMPI itself, and therefore mpirun, then you can 
>> build the various components if you first install the headers for that 
>> environment on your system. It means that you need one machine where all 
>> those resource managers at least have their headers installed on it. Then 
>> configure OMPI --with-xxx pointing to each of the RM’s headers so all the 
>> components get built. When the binary hits your customer’s machine, only 
>> those components that have active libraries present will execute.
>>  
>>  
>> 2.
>> For integration with LSF/grid, how would I specify the memory (RAM) 
>> requirement (or some other parameter) to bsub/qsub, when launching mpirun 
>> command? Will something like below work to ensure that each of the 8 copies 
>> of a.out have 40 GB memory reserved for them by grid engine?
>>  
>> qsub –pe orte 8 –b y –V –l m_mem_free=40G –cwd mpirun –np 8 a.out
>>  
>> You’ll have to provide something that is environment dependent, I’m afraid - 
>> there is no standard out there.
>> 
>> 
>>  
>> 3.
>> Some of our customers use custom distribution engine (some 
>> non-industry-standard distribution engine). How can I integrate my openmpi  
>> application with such system? I would think that it should be possible to do 
>> that if openmpi launched/managed interaction with the distribution engine 
>> using some kind of generic mechanism (say, use a configurable command to 
>> launch, monitor, kill a job and then allow specification of a plugin define 
>> these operations with commands specific to the distribution engine being in 
>> use). Does such integration exist in openmpi?
>>  
>> Easiest solution is to write a script that reads the allocation and dumps it 
>> into a file, and then provide that file as your hostfile on the mpirun cmd 
>> line (or in the environment). We will then use ssh to perform the launch. 
>> Otherwise, you’ll need to write at least an orte/mca/ras component to get 
>> the 

Re: [OMPI users] Questions about integration with resource distribution systems

[Reuti]

Hi,

> Am 26.07.2017 um 15:03 schrieb Kulshrestha, Vipul 
> :
> 
> Thanks for a quick response.
>  
> I will try building OMPI as suggested.
>  
> On the integration with unsupported distribution systems, we cannot use 
> script based approach, because often these machines don’t have ssh permission 
> in customer environment. I will explore the path of writing orte component. 
> At this stage, I don’t understand the effort for the same.
>  
> I guess my question 2 was not understood correctly. I used the below command 
> as an example for SGE and want to understand the expected behavior for such a 
> command. With the below command, I expect to have 8 copies of a.out launched

Yep.


> with each copy having access to 40GB of memory. Is that correct?

SGE will grant the memory, not Open MPI. This is done by SGE's tight 
integration and as slave tasks are started by `qrsh -inherit …` and not by a 
plain `ssh`. This way SGE can keep track of the started processes.


> I am doubtful, because I don’t understand how mpirun gets access to 
> information about RAM requirement.
>  
> qsub –pe orte 8 –b y –V –l m_mem_free=40G –cwd mpirun –np 8 a.out

In case your application relies on the actual value of "m_mem_free", your 
application has to request this information. This might different between the 
various queuing systems though. In SGE one could either use `qstat` to `grep` 
the information, or (instead of a direct `mpirun`) uses a jobscript which will 
feed this value in addition to an environment variable, which you can access 
directly in your application. On a command line it would be:

$ qsub –pe orte 8 –b y –v m_mem_free=40G –l m_mem_free=40G –cwd mpirun –np 8 
a.out

1. -V might set to many variable. Usually I suggest to forward only environment 
variables which are necessary for the job. The user could set some environment 
variable by accident and wonder why the job, which started a couple of days 
later only, crashes; but submitting exactly the same job again succeeds.

2. The 40G is a string in the environment variable, you may want to use the 
plain value in bytes there.

-- Reuti
___
users mailing list
users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

Re: [OMPI users] Questions about integration with resource distribution systems

[r...@open-mpi.org]

mpirun doesn’t get access to that requirement, nor does it need to do so. SGE 
will use the requirement when determining the nodes to allocate. mpirun just 
uses the nodes that SGE provides.

What your cmd line does is restrict the entire operation on each node (daemon + 
8 procs) to 40GB of memory. OMPI does not support per-process restrictions 
other than binding to cpus.


> On Jul 26, 2017, at 6:03 AM, Kulshrestha, Vipul 
>  wrote:
> 
> Thanks for a quick response.
>  
> I will try building OMPI as suggested.
>  
> On the integration with unsupported distribution systems, we cannot use 
> script based approach, because often these machines don’t have ssh permission 
> in customer environment. I will explore the path of writing orte component. 
> At this stage, I don’t understand the effort for the same.
>  
> I guess my question 2 was not understood correctly. I used the below command 
> as an example for SGE and want to understand the expected behavior for such a 
> command. With the below command, I expect to have 8 copies of a.out launched 
> with each copy having access to 40GB of memory. Is that correct? I am 
> doubtful, because I don’t understand how mpirun gets access to information 
> about RAM requirement.
>  
> qsub –pe orte 8 –b y –V –l m_mem_free=40G –cwd mpirun –np 8 a.out
>  
>  
> Regards,
> Vipul
>  
>  
>   <>
> From: users [mailto:users-boun...@lists.open-mpi.org] On Behalf Of 
> r...@open-mpi.org
> Sent: Tuesday, July 25, 2017 8:16 PM
> To: Open MPI Users 
> Subject: Re: [OMPI users] Questions about integration with resource 
> distribution systems
>  
>  
> On Jul 25, 2017, at 3:48 PM, Kulshrestha, Vipul  > wrote:
>  
> I have several questions about integration of openmpi with resource queuing 
> systems.
>  
> 1.
> I understand that openmpi supports integration with various resource 
> distribution systems such as SGE, LSF, torque etc.
>  
> I need to build an openmpi application that can interact with variety of 
> different resource distribution systems, since different customers have 
> different systems. Based on my research, it seems that I need to build a 
> different openmpi installation to work, e.g. create an installation of opempi 
> with grid and create a different installation of openmpi with LSF. Is there a 
> way to build a generic installation of openmpi that can be used with more 
> than 1 distribution system by using some generic mechanism?
>  
> Just to be clear: your application doesn’t depend on the environment in any 
> way. Only mpirun does - so if you are distributing an _application_, then 
> your question is irrelevant. 
>  
> If you are distributing OMPI itself, and therefore mpirun, then you can build 
> the various components if you first install the headers for that environment 
> on your system. It means that you need one machine where all those resource 
> managers at least have their headers installed on it. Then configure OMPI 
> --with-xxx pointing to each of the RM’s headers so all the components get 
> built. When the binary hits your customer’s machine, only those components 
> that have active libraries present will execute.
>  
>  
> 2.
> For integration with LSF/grid, how would I specify the memory (RAM) 
> requirement (or some other parameter) to bsub/qsub, when launching mpirun 
> command? Will something like below work to ensure that each of the 8 copies 
> of a.out have 40 GB memory reserved for them by grid engine?
>  
> qsub –pe orte 8 –b y –V –l m_mem_free=40G –cwd mpirun –np 8 a.out
>  
> You’ll have to provide something that is environment dependent, I’m afraid - 
> there is no standard out there.
> 
> 
>  
> 3.
> Some of our customers use custom distribution engine (some 
> non-industry-standard distribution engine). How can I integrate my openmpi  
> application with such system? I would think that it should be possible to do 
> that if openmpi launched/managed interaction with the distribution engine 
> using some kind of generic mechanism (say, use a configurable command to 
> launch, monitor, kill a job and then allow specification of a plugin define 
> these operations with commands specific to the distribution engine being in 
> use). Does such integration exist in openmpi?
>  
> Easiest solution is to write a script that reads the allocation and dumps it 
> into a file, and then provide that file as your hostfile on the mpirun cmd 
> line (or in the environment). We will then use ssh to perform the launch. 
> Otherwise, you’ll need to write at least an orte/mca/ras component to get the 
> allocation, and possibly an orte/mca/plm component if you want to use the 
> native launch mechanism in place of ssh.
> 
> 
>  
>  
> Thanks,
> Vipul
>  
>  
> ___
> users mailing list
> users@lists.open-mpi.org 
> 

Re: [OMPI users] Groups and Communicators

[George Bosilca]

Diego,

As all your processes are started under the umbrella of a single mpirun,
they have a communicator in common, the MPI_COMM_WORLD.

One possible implementation, using MPI_Comm_split, will be the following:

MPI_Comm small_comm, leader_comm;

/* Create small_comm on all processes */

/* Now use MPI_Comm_split on MPI_COMM_WORLD to select the leaders */
MPI_Comm_split( MPI_COMM_WORLD,
 i_am_leader(small_comm) ? 1 : MPI_UNDEFINED,
 rank_in_comm_world,
 _Comm);

The leader_comm will be a valid communicator on all leaders processes, and
MPI_COMM_NULL on all others.

  George.



On Wed, Jul 26, 2017 at 4:29 AM, Diego Avesani 
wrote:

> Dear George, Dear all,
>
> I use "mpirun -np xx ./a.out"
>
> I do not know if I have some common  grounds. I mean, I have to
> design everything from the begging. You can find what I would like to do in
> the attachment. Basically, an MPI cast in another MPI. Consequently, I am
> thinking to MPI groups or MPI virtual topology with a 2D cart, using the
> columns as "groups" and the first rows as the external groups to handle the
> columns.
>
> What do think? What do you suggest?
> Really Really thanks
>
>
> Diego
>
>
> On 25 July 2017 at 19:26, George Bosilca  wrote:
>
>> Diego,
>>
>> Assuming you have some common  grounds between the 4 initial groups
>> (otherwise you will have to connect them via 
>> MPI_Comm_connect/MPI_Comm_accept)
>> you can merge the 4 groups together and then use any MPI mechanism to
>> create a partial group of leaders (such as MPI_Comm_split).
>>
>> If you spawn the groups via MPI_Comm_spawn then the answer is slightly
>> more complicated, you need to use MPI_Intercomm_create, with the spawner as
>> the bridge between the different communicators (and then
>> MPI_Intercomm_merge to create your intracomm). You can find a good answer
>> on stackoverflow on this at https://stackoverflow.com/ques
>> tions/24806782/mpi-merge-multiple-intercoms-into-a-single-intracomm
>>
>> How is your MPI environment started (single mpirun or mpi_comm_spawn) ?
>>
>>   George.
>>
>>
>>
>> On Tue, Jul 25, 2017 at 10:44 AM, Diego Avesani 
>> wrote:
>>
>>> Dear All,
>>>
>>> I am studying Groups and Communicators, but before start going in
>>> detail, I have a question about groups.
>>>
>>> I would like to know if is it possible to create a group of masters of
>>> the other groups and then a intra-communication in the new group. I have
>>> spent sometime reading different tutorial and presentation, but it is
>>> difficult, at least for me, to understand if is it possible to create this
>>> sort of MPI cast in another MPI.
>>>
>>> In the attachment you can find a pictures that summarize what I would
>>> like to do.
>>>
>>> Another strategies could be use virtual topology.
>>>
>>> What do you think?
>>>
>>> I really, really, appreciate any kind of help, suggestions or link where
>>> I can study this topics.
>>>
>>> Again, thanks
>>>
>>> Best Regards,
>>>
>>> Diego
>>>
>>> ___
>>> users mailing list
>>> users@lists.open-mpi.org
>>> https://rfd.newmexicoconsortium.org/mailman/listinfo/users
>>>
>>
>>
>> ___
>> users mailing list
>> users@lists.open-mpi.org
>> https://rfd.newmexicoconsortium.org/mailman/listinfo/users
>>
>
>
> ___
> users mailing list
> users@lists.open-mpi.org
> https://rfd.newmexicoconsortium.org/mailman/listinfo/users
>
___
users mailing list
users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

Re: [OMPI users] Questions about integration with resource distribution systems

[Kulshrestha, Vipul]

Thanks for a quick response.

I will try building OMPI as suggested.

On the integration with unsupported distribution systems, we cannot use script 
based approach, because often these machines don’t have ssh permission in 
customer environment. I will explore the path of writing orte component. At 
this stage, I don’t understand the effort for the same.

I guess my question 2 was not understood correctly. I used the below command as 
an example for SGE and want to understand the expected behavior for such a 
command. With the below command, I expect to have 8 copies of a.out launched 
with each copy having access to 40GB of memory. Is that correct? I am doubtful, 
because I don’t understand how mpirun gets access to information about RAM 
requirement.

qsub –pe orte 8 –b y –V –l m_mem_free=40G –cwd mpirun –np 8 a.out


Regards,
Vipul



From: users [mailto:users-boun...@lists.open-mpi.org] On Behalf Of 
r...@open-mpi.org
Sent: Tuesday, July 25, 2017 8:16 PM
To: Open MPI Users 
Subject: Re: [OMPI users] Questions about integration with resource 
distribution systems


On Jul 25, 2017, at 3:48 PM, Kulshrestha, Vipul 
> wrote:

I have several questions about integration of openmpi with resource queuing 
systems.

1.
I understand that openmpi supports integration with various resource 
distribution systems such as SGE, LSF, torque etc.

I need to build an openmpi application that can interact with variety of 
different resource distribution systems, since different customers have 
different systems. Based on my research, it seems that I need to build a 
different openmpi installation to work, e.g. create an installation of opempi 
with grid and create a different installation of openmpi with LSF. Is there a 
way to build a generic installation of openmpi that can be used with more than 
1 distribution system by using some generic mechanism?

Just to be clear: your application doesn’t depend on the environment in any 
way. Only mpirun does - so if you are distributing an _application_, then your 
question is irrelevant.

If you are distributing OMPI itself, and therefore mpirun, then you can build 
the various components if you first install the headers for that environment on 
your system. It means that you need one machine where all those resource 
managers at least have their headers installed on it. Then configure OMPI 
--with-xxx pointing to each of the RM’s headers so all the components get 
built. When the binary hits your customer’s machine, only those components that 
have active libraries present will execute.


2.
For integration with LSF/grid, how would I specify the memory (RAM) requirement 
(or some other parameter) to bsub/qsub, when launching mpirun command? Will 
something like below work to ensure that each of the 8 copies of a.out have 40 
GB memory reserved for them by grid engine?

qsub –pe orte 8 –b y –V –l m_mem_free=40G –cwd mpirun –np 8 a.out

You’ll have to provide something that is environment dependent, I’m afraid - 
there is no standard out there.



3.
Some of our customers use custom distribution engine (some 
non-industry-standard distribution engine). How can I integrate my openmpi  
application with such system? I would think that it should be possible to do 
that if openmpi launched/managed interaction with the distribution engine using 
some kind of generic mechanism (say, use a configurable command to launch, 
monitor, kill a job and then allow specification of a plugin define these 
operations with commands specific to the distribution engine being in use). 
Does such integration exist in openmpi?

Easiest solution is to write a script that reads the allocation and dumps it 
into a file, and then provide that file as your hostfile on the mpirun cmd line 
(or in the environment). We will then use ssh to perform the launch. Otherwise, 
you’ll need to write at least an orte/mca/ras component to get the allocation, 
and possibly an orte/mca/plm component if you want to use the native launch 
mechanism in place of ssh.




Thanks,
Vipul


___
users mailing list
users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

___
users mailing list
users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

Re: [OMPI users] MPI_IN_PLACE

[Gilles Gouaillardet]

Volker,

thanks, i will have a look at it

meanwhile, if you can reproduce this issue on a more mainstream
platform (e.g. linux + gfortran) please let me know.

since you are using ifort, Open MPI was built with Fortran 2008
bindings, so you can replace
include 'mpif.h'
with
use mpi_f08
and who knows, that might solve your issue


Cheers,

Gilles

On Wed, Jul 26, 2017 at 5:22 PM, Volker Blum  wrote:
> Dear Gilles,
>
> Thank you very much for the fast answer.
>
> Darn. I feared it might not occur on all platforms, since my former Macbook
> (with an older OpenMPI version) no longer exhibited the problem, a different
> Linux/Intel Machine did last December, etc.
>
> On this specific machine, the configure line is
>
> ./configure CC=gcc FC=ifort F77=ifort
>
> ifort version 17.0.4
>
> blum:/Users/blum/software/openmpi-3.0.0rc1> gcc -v
> Configured with: --prefix=/Applications/Xcode.app/Contents/Developer/usr
> --with-gxx-include-dir=/usr/include/c++/4.2.1
> Apple LLVM version 8.1.0 (clang-802.0.42)
> Target: x86_64-apple-darwin16.6.0
> Thread model: posix
> InstalledDir:
> /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin
>
> The full test program is appended.
>
> Compilation:
>
> mpif90 check_mpi_in_place.f90
>
> blum:/Users/blum/codes/fhi-aims/openmpi_test> which mpif90
> /usr/local/openmpi-3.0.0rc1/bin/mpif90
>
> blum:/Users/blum/codes/fhi-aims/openmpi_test> which mpirun
> /usr/local/openmpi-3.0.0rc1/bin/mpirun
>
> blum:/Users/blum/codes/fhi-aims/openmpi_test> mpirun -np 2 a.out
>  * MPI_IN_PLACE does not appear to work as intended.
>  * Checking whether MPI_ALLREDUCE works at all.
>  * Without MPI_IN_PLACE, MPI_ALLREDUCE appears to work.
>
> blum:/Users/blum/codes/fhi-aims/openmpi_test> mpirun -np 1 a.out
>  * MPI_IN_PLACE does not appear to work as intended.
>  * Checking whether MPI_ALLREDUCE works at all.
>  * Without MPI_IN_PLACE, MPI_ALLREDUCE appears to work.
>
> Hopefully, no trivial mistakes in the testcase. I just spent a few days
> tracing this issue through a fairly large code, which is where the issue
> originally arose (and leads to wrong numbers).
>
> Best wishes
> Volker
>
>
>
>
>> On Jul 26, 2017, at 9:46 AM, Gilles Gouaillardet
>>  wrote:
>>
>> Volker,
>>
>> i was unable to reproduce this issue on linux
>>
>> can you please post your full configure command line, your gnu
>> compiler version and the full test program ?
>>
>> also, how many mpi tasks are you running ?
>>
>> Cheers,
>>
>> Gilles
>>
>> On Wed, Jul 26, 2017 at 4:25 PM, Volker Blum  wrote:
>>> Hi,
>>>
>>> I tried openmpi-3.0.0rc1.tar.gz using Intel Fortran 2017 and gcc on a
>>> current MacOS system. For this version, it seems to me that MPI_IN_PLACE
>>> returns incorrect results (while other MPI implementations, including some
>>> past OpenMPI versions, work fine).
>>>
>>> This can be seen with a simple Fortran example code, shown below. In the
>>> test, the values of all entries of an array “test_data” should be 1.0d0 if
>>> the behavior were as intended. However, the version of OpenMPI I have
>>> returns 0.d0 instead.
>>>
>>> I’ve seen this behavior on some other compute platforms too, in the past,
>>> so it wasn’t new to me. Still, I thought that this time, I’d ask. Any
>>> thoughts?
>>>
>>> Thank you,
>>> Best wishes
>>> Volker
>>>
>>>! size of test data array
>>>integer :: n_data
>>>
>>>! array that contains test data for MPI_IN_PLACE
>>>real*8, allocatable :: test_data(:)
>>>
>>>integer :: mpierr
>>>
>>>n_data = 10
>>>
>>>allocate(test_data(n_data),stat=mpierr)
>>>
>>>! seed test data array for allreduce call below
>>>if (myid.eq.0) then
>>>   test_data(:) = 1.d0
>>>else
>>>   test_data(:) = 0.d0
>>>end if
>>>
>>>! Sum the test_data array over all MPI tasks
>>>call MPI_ALLREDUCE(MPI_IN_PLACE, &
>>> test_data(:), &
>>> n_data, &
>>> MPI_DOUBLE_PRECISION, &
>>> MPI_SUM, &
>>> mpi_comm_global, &
>>> mpierr )
>>>
>>>! The value of all entries of test_data should now be 1.d0 on all MPI
>>> tasks.
>>>! If that is not the case, then the MPI_IN_PLACE flag may be broken.
>>>
>>>
>>>
>>>
>>>
>>>
>>> Volker Blum
>>> Associate Professor
>>> Ab Initio Materials Simulations
>>> Duke University, MEMS Department
>>> 144 Hudson Hall, Box 90300, Duke University, Durham, NC 27708, USA
>>>
>>> volker.b...@duke.edu
>>> https://aims.pratt.duke.edu
>>> +1 (919) 660 5279
>>> Twitter: Aimsduke
>>>
>>> Office:  Hudson Hall
>>>
>>>
>>>
>>>
>>> ___
>>> users mailing list
>>> users@lists.open-mpi.org
>>>
>>> 

Re: [OMPI users] Questions about integration with resource distribution systems

[Reuti]

Hi,

> Am 26.07.2017 um 02:16 schrieb r...@open-mpi.org:
> 
> 
>> On Jul 25, 2017, at 3:48 PM, Kulshrestha, Vipul 
>>  wrote:
>> 
>> I have several questions about integration of openmpi with resource queuing 
>> systems.
>>  
>> 1.
>> I understand that openmpi supports integration with various resource 
>> distribution systems such as SGE, LSF, torque etc.
>>  
>> I need to build an openmpi application that can interact with variety of 
>> different resource distribution systems, since different customers have 
>> different systems. Based on my research, it seems that I need to build a 
>> different openmpi installation to work, e.g. create an installation of 
>> opempi with grid and create a different installation of openmpi with LSF. Is 
>> there a way to build a generic installation of openmpi that can be used with 
>> more than 1 distribution system by using some generic mechanism?
> 
> Just to be clear: your application doesn’t depend on the environment in any 
> way. Only mpirun does - so if you are distributing an _application_, then 
> your question is irrelevant. 
> 
> If you are distributing OMPI itself, and therefore mpirun, then you can build 
> the various components if you first install the headers for that environment 
> on your system. It means that you need one machine where all those resource 
> managers at least have their headers installed on it. Then configure OMPI 
> --with-xxx pointing to each of the RM’s headers so all the components get 
> built. When the binary hits your customer’s machine, only those components 
> that have active libraries present will execute.

Just note, that the SGE integration doesn't need any library. It's invoked when 
Open MPI finds certain environment variables set to sensitive values.

-- Reuti
___
users mailing list
users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

Re: [OMPI users] Questions about integration with resource distribution systems

[Reuti]

Hi,

> Am 26.07.2017 um 00:48 schrieb Kulshrestha, Vipul 
> :
> 
> I have several questions about integration of openmpi with resource queuing 
> systems.
>  
> 1.
> I understand that openmpi supports integration with various resource 
> distribution systems such as SGE, LSF, torque etc.
>  
> I need to build an openmpi application that can interact with variety of 
> different resource distribution systems, since different customers have 
> different systems. Based on my research, it seems that I need to build a 
> different openmpi installation to work, e.g. create an installation of opempi 
> with grid and create a different installation of openmpi with LSF. Is there a 
> way to build a generic installation of openmpi that can be used with more 
> than 1 distribution system by using some generic mechanism?
>  
> 2.
> For integration with LSF/grid, how would I specify the memory (RAM) 
> requirement (or some other parameter) to bsub/qsub, when launching mpirun 
> command? Will something like below work to ensure that each of the 8 copies 
> of a.out have 40 GB memory reserved for them by grid engine?
>  
> qsub –pe orte 8 –b y –V –l m_mem_free=40G –cwd mpirun –np 8 a.out

m_mem_free is part of Univa SGE (but not the various free ones of SGE AFAIK).

Also: this syntax is for SGE, in LSF it's different.

To have this independent from the actual queuing system, one could look into 
DRMAA V.2 (unfortunately not many are supporting this version of DRMAA).

-- Reuti
___
users mailing list
users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

Re: [OMPI users] Groups and Communicators

[Diego Avesani]

Dear George, Dear all,

I use "mpirun -np xx ./a.out"

I do not know if I have some common  grounds. I mean, I have to
design everything from the begging. You can find what I would like to do in
the attachment. Basically, an MPI cast in another MPI. Consequently, I am
thinking to MPI groups or MPI virtual topology with a 2D cart, using the
columns as "groups" and the first rows as the external groups to handle the
columns.

What do think? What do you suggest?
Really Really thanks


Diego


On 25 July 2017 at 19:26, George Bosilca  wrote:

> Diego,
>
> Assuming you have some common  grounds between the 4 initial groups
> (otherwise you will have to connect them via MPI_Comm_connect/MPI_Comm_accept)
> you can merge the 4 groups together and then use any MPI mechanism to
> create a partial group of leaders (such as MPI_Comm_split).
>
> If you spawn the groups via MPI_Comm_spawn then the answer is slightly
> more complicated, you need to use MPI_Intercomm_create, with the spawner as
> the bridge between the different communicators (and then
> MPI_Intercomm_merge to create your intracomm). You can find a good answer
> on stackoverflow on this at https://stackoverflow.com/
> questions/24806782/mpi-merge-multiple-intercoms-into-a-single-intracomm
>
> How is your MPI environment started (single mpirun or mpi_comm_spawn) ?
>
>   George.
>
>
>
> On Tue, Jul 25, 2017 at 10:44 AM, Diego Avesani 
> wrote:
>
>> Dear All,
>>
>> I am studying Groups and Communicators, but before start going in detail,
>> I have a question about groups.
>>
>> I would like to know if is it possible to create a group of masters of
>> the other groups and then a intra-communication in the new group. I have
>> spent sometime reading different tutorial and presentation, but it is
>> difficult, at least for me, to understand if is it possible to create this
>> sort of MPI cast in another MPI.
>>
>> In the attachment you can find a pictures that summarize what I would
>> like to do.
>>
>> Another strategies could be use virtual topology.
>>
>> What do you think?
>>
>> I really, really, appreciate any kind of help, suggestions or link where
>> I can study this topics.
>>
>> Again, thanks
>>
>> Best Regards,
>>
>> Diego
>>
>> ___
>> users mailing list
>> users@lists.open-mpi.org
>> https://rfd.newmexicoconsortium.org/mailman/listinfo/users
>>
>
>
> ___
> users mailing list
> users@lists.open-mpi.org
> https://rfd.newmexicoconsortium.org/mailman/listinfo/users
>


what2do.pdf
Description: Adobe PDF document
___
users mailing list
users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

Re: [OMPI users] MPI_IN_PLACE

[Volker Blum]

Dear Gilles,

Thank you very much for the fast answer.

Darn. I feared it might not occur on all platforms, since my former Macbook 
(with an older OpenMPI version) no longer exhibited the problem, a different 
Linux/Intel Machine did last December, etc.

On this specific machine, the configure line is

./configure CC=gcc FC=ifort F77=ifort

ifort version 17.0.4

blum:/Users/blum/software/openmpi-3.0.0rc1> gcc -v
Configured with: --prefix=/Applications/Xcode.app/Contents/Developer/usr 
--with-gxx-include-dir=/usr/include/c++/4.2.1
Apple LLVM version 8.1.0 (clang-802.0.42)
Target: x86_64-apple-darwin16.6.0
Thread model: posix
InstalledDir: 
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin

The full test program is appended.

Compilation:

mpif90 check_mpi_in_place.f90

blum:/Users/blum/codes/fhi-aims/openmpi_test> which mpif90
/usr/local/openmpi-3.0.0rc1/bin/mpif90

blum:/Users/blum/codes/fhi-aims/openmpi_test> which mpirun
/usr/local/openmpi-3.0.0rc1/bin/mpirun

blum:/Users/blum/codes/fhi-aims/openmpi_test> mpirun -np 2 a.out
 * MPI_IN_PLACE does not appear to work as intended.
 * Checking whether MPI_ALLREDUCE works at all.
 * Without MPI_IN_PLACE, MPI_ALLREDUCE appears to work.

blum:/Users/blum/codes/fhi-aims/openmpi_test> mpirun -np 1 a.out
 * MPI_IN_PLACE does not appear to work as intended.
 * Checking whether MPI_ALLREDUCE works at all.
 * Without MPI_IN_PLACE, MPI_ALLREDUCE appears to work.

Hopefully, no trivial mistakes in the testcase. I just spent a few days tracing 
this issue through a fairly large code, which is where the issue originally 
arose (and leads to wrong numbers).

Best wishes
Volker




> On Jul 26, 2017, at 9:46 AM, Gilles Gouaillardet 
>  wrote:
>
> Volker,
>
> i was unable to reproduce this issue on linux
>
> can you please post your full configure command line, your gnu
> compiler version and the full test program ?
>
> also, how many mpi tasks are you running ?
>
> Cheers,
>
> Gilles
>
> On Wed, Jul 26, 2017 at 4:25 PM, Volker Blum  wrote:
>> Hi,
>>
>> I tried openmpi-3.0.0rc1.tar.gz using Intel Fortran 2017 and gcc on a 
>> current MacOS system. For this version, it seems to me that MPI_IN_PLACE 
>> returns incorrect results (while other MPI implementations, including some 
>> past OpenMPI versions, work fine).
>>
>> This can be seen with a simple Fortran example code, shown below. In the 
>> test, the values of all entries of an array “test_data” should be 1.0d0 if 
>> the behavior were as intended. However, the version of OpenMPI I have 
>> returns 0.d0 instead.
>>
>> I’ve seen this behavior on some other compute platforms too, in the past, so 
>> it wasn’t new to me. Still, I thought that this time, I’d ask. Any thoughts?
>>
>> Thank you,
>> Best wishes
>> Volker
>>
>>! size of test data array
>>integer :: n_data
>>
>>! array that contains test data for MPI_IN_PLACE
>>real*8, allocatable :: test_data(:)
>>
>>integer :: mpierr
>>
>>n_data = 10
>>
>>allocate(test_data(n_data),stat=mpierr)
>>
>>! seed test data array for allreduce call below
>>if (myid.eq.0) then
>>   test_data(:) = 1.d0
>>else
>>   test_data(:) = 0.d0
>>end if
>>
>>! Sum the test_data array over all MPI tasks
>>call MPI_ALLREDUCE(MPI_IN_PLACE, &
>> test_data(:), &
>> n_data, &
>> MPI_DOUBLE_PRECISION, &
>> MPI_SUM, &
>> mpi_comm_global, &
>> mpierr )
>>
>>! The value of all entries of test_data should now be 1.d0 on all MPI 
>> tasks.
>>! If that is not the case, then the MPI_IN_PLACE flag may be broken.
>>
>>
>>
>>
>>
>>
>> Volker Blum
>> Associate Professor
>> Ab Initio Materials Simulations
>> Duke University, MEMS Department
>> 144 Hudson Hall, Box 90300, Duke University, Durham, NC 27708, USA
>>
>> volker.b...@duke.edu
>> https://aims.pratt.duke.edu
>> +1 (919) 660 5279
>> Twitter: Aimsduke
>>
>> Office:  Hudson Hall
>>
>>
>>
>>
>> ___
>> users mailing list
>> users@lists.open-mpi.org
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__rfd.newmexicoconsortium.org_mailman_listinfo_users=DwIGaQ=imBPVzF25OnBgGmVOlcsiEgHoG1i6YHLR0Sj_gZ4adc=I9QLPu689VeINkpRod6EQfprr_v-FLoLsAuSXIHhDsk=QLtQXqnGSkgQnmgI4RxZXa9R6FhMmgj2FLN452Q0Wis=BeracGkSHhIyI_bjKJqPHCqMuP-Se2pRmbiNfugkdK8=
> ___
> users mailing list
> users@lists.open-mpi.org
> https://urldefense.proofpoint.com/v2/url?u=https-3A__rfd.newmexicoconsortium.org_mailman_listinfo_users=DwIGaQ=imBPVzF25OnBgGmVOlcsiEgHoG1i6YHLR0Sj_gZ4adc=I9QLPu689VeINkpRod6EQfprr_v-FLoLsAuSXIHhDsk=QLtQXqnGSkgQnmgI4RxZXa9R6FhMmgj2FLN452Q0Wis=BeracGkSHhIyI_bjKJqPHCqMuP-Se2pRmbiNfugkdK8=

Volker Blum
Associate Professor
Ab Initio Materials Simulations
Duke University, MEMS Department
144 Hudson Hall, Box 90300, Duke University, Durham, NC 27708, USA

volker.b...@duke.edu

Re: [OMPI users] MPI_IN_PLACE

[Gilles Gouaillardet]

Volker,

i was unable to reproduce this issue on linux

can you please post your full configure command line, your gnu
compiler version and the full test program ?

also, how many mpi tasks are you running ?

Cheers,

Gilles

On Wed, Jul 26, 2017 at 4:25 PM, Volker Blum  wrote:
> Hi,
>
> I tried openmpi-3.0.0rc1.tar.gz using Intel Fortran 2017 and gcc on a current 
> MacOS system. For this version, it seems to me that MPI_IN_PLACE returns 
> incorrect results (while other MPI implementations, including some past 
> OpenMPI versions, work fine).
>
> This can be seen with a simple Fortran example code, shown below. In the 
> test, the values of all entries of an array “test_data” should be 1.0d0 if 
> the behavior were as intended. However, the version of OpenMPI I have returns 
> 0.d0 instead.
>
> I’ve seen this behavior on some other compute platforms too, in the past, so 
> it wasn’t new to me. Still, I thought that this time, I’d ask. Any thoughts?
>
> Thank you,
> Best wishes
> Volker
>
> ! size of test data array
> integer :: n_data
>
> ! array that contains test data for MPI_IN_PLACE
> real*8, allocatable :: test_data(:)
>
> integer :: mpierr
>
> n_data = 10
>
> allocate(test_data(n_data),stat=mpierr)
>
> ! seed test data array for allreduce call below
> if (myid.eq.0) then
>test_data(:) = 1.d0
> else
>test_data(:) = 0.d0
> end if
>
> ! Sum the test_data array over all MPI tasks
> call MPI_ALLREDUCE(MPI_IN_PLACE, &
>  test_data(:), &
>  n_data, &
>  MPI_DOUBLE_PRECISION, &
>  MPI_SUM, &
>  mpi_comm_global, &
>  mpierr )
>
> ! The value of all entries of test_data should now be 1.d0 on all MPI 
> tasks.
> ! If that is not the case, then the MPI_IN_PLACE flag may be broken.
>
>
>
>
>
>
> Volker Blum
> Associate Professor
> Ab Initio Materials Simulations
> Duke University, MEMS Department
> 144 Hudson Hall, Box 90300, Duke University, Durham, NC 27708, USA
>
> volker.b...@duke.edu
> https://aims.pratt.duke.edu
> +1 (919) 660 5279
> Twitter: Aimsduke
>
> Office:  Hudson Hall
>
>
>
>
> ___
> users mailing list
> users@lists.open-mpi.org
> https://rfd.newmexicoconsortium.org/mailman/listinfo/users
___
users mailing list
users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

[OMPI users] MPI_IN_PLACE

[Volker Blum]

Hi,

I tried openmpi-3.0.0rc1.tar.gz using Intel Fortran 2017 and gcc on a current 
MacOS system. For this version, it seems to me that MPI_IN_PLACE returns 
incorrect results (while other MPI implementations, including some past OpenMPI 
versions, work fine).

This can be seen with a simple Fortran example code, shown below. In the test, 
the values of all entries of an array “test_data” should be 1.0d0 if the 
behavior were as intended. However, the version of OpenMPI I have returns 0.d0 
instead.

I’ve seen this behavior on some other compute platforms too, in the past, so it 
wasn’t new to me. Still, I thought that this time, I’d ask. Any thoughts?

Thank you,
Best wishes
Volker

! size of test data array
integer :: n_data
  
! array that contains test data for MPI_IN_PLACE
real*8, allocatable :: test_data(:)

integer :: mpierr

n_data = 10

allocate(test_data(n_data),stat=mpierr)

! seed test data array for allreduce call below
if (myid.eq.0) then
   test_data(:) = 1.d0
else
   test_data(:) = 0.d0
end if

! Sum the test_data array over all MPI tasks
call MPI_ALLREDUCE(MPI_IN_PLACE, &
 test_data(:), &
 n_data, &
 MPI_DOUBLE_PRECISION, &
 MPI_SUM, &
 mpi_comm_global, &
 mpierr )

! The value of all entries of test_data should now be 1.d0 on all MPI tasks.
! If that is not the case, then the MPI_IN_PLACE flag may be broken.






Volker Blum
Associate Professor
Ab Initio Materials Simulations
Duke University, MEMS Department 
144 Hudson Hall, Box 90300, Duke University, Durham, NC 27708, USA

volker.b...@duke.edu
https://aims.pratt.duke.edu
+1 (919) 660 5279
Twitter: Aimsduke

Office:  Hudson Hall




___
users mailing list
users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/users