[OMPI devel] MPI ABI effort

[Gilles Gouaillardet via devel]

Folks,

Jeff Hammond and al. published "MPI Application Binary Interface
Standardization" las week
https://arxiv.org/abs/2308.11214

The paper reads the (C) ABI has already been prototyped natively in MPICH.

Is there any current interest into prototyping this ABI into Open MPI?


Cheers,

Gilles

Re: [OMPI devel] Open MPI v5.0.0rc9 is available for testing

[Gilles Gouaillardet via devel]

Well, --enable-mca-no-build=io-romio341 works strictly speaking (it does
not build the io/romio341 component).

That being said, it does not prevent 3rd-party/romio from being built, and
this is what fails with gcc 4.8

I will file an issue in order to keep track of that.


Cheers,

Gilles

On Fri, Nov 18, 2022 at 9:07 AM Barrett, Brian via devel <
devel@lists.open-mpi.org> wrote:

> --enable-mca-no-build=io-romio341 should still work.  Or just
> --disable-io-romio.
>
> No comment around the RHEL7 part; that's pretty old, but I don't think
> we've officially said it is too old.  Probably something worth filing a
> ticket for so that we can run to ground before 5.0 release.  Oddly, CI
> continues to pass on RHEL7 in AWS.  I'm not sure what we've done to cause
> that, but also worth investigating.
>
> Brian
>
> On 11/14/22, 10:32 PM, "devel on behalf of Gilles Gouaillardet via devel"
> 
> wrote:
>
> CAUTION: This email originated from outside of the organization. Do
> not click links or open attachments unless you can confirm the sender and
> know the content is safe.
>
>
>
> Folks,
>
>
> I tried to build on a RHEL7 like, and it fails at make time because
> ROMIO requires stdatomic.h
>
> (it seems this file is only available from GCC 4.9)
>
>
> Are we supposed to be able to build Open MPI 5 with GCC 4.8 (e.g. stock
> RHEL7 compiler)?
>
>
> --enable-mca-no-build=io-romio314 cannot help since ROMIO has been
> moved
> to 3rd-party
>
> is there an other workaround? if not, should be add something like
> --enable-3rd-party-no-build=romio?
>
>
> Cheers,
>
>
> Gilles
>
> On 10/22/2022 6:23 AM, Austen W Lauria via devel wrote:
> > Open MPI v5.0.0rc9 is now available for testing
> > athttps://www.open-mpi.org/software/ompi/v5.0/
> > <https://www.open-mpi.org/software/ompi/v5.0/>.
> >
> > Please test, and send feedback either via the user mailing lists, or
> > create an issue athttps://github.com/open-mpi/ompi/issues/
> > <https://github.com/open-mpi/ompi/issues/> .
> >
> > Seehttps://docs.open-mpi.org/en/v5.0.x/news/news-v5.0.x.html
> > <https://docs.open-mpi.org/en/v5.0.x/news/news-v5.0.x.html> for a
> list
> > of changes since rc8.
> >
> > Thank you,
> > v5.0 Release Managers
>
>
>

Re: [OMPI devel] Open MPI v5.0.0rc9 is available for testing

[Gilles Gouaillardet via devel]

This is very odd ...

I dumped DISABLE_io_romio341, and it is *not* set (!)

Anyway, let's follow-up at https://github.com/open-mpi/ompi/issues/11088

Cheers,

Gilles

On Fri, Nov 18, 2022 at 5:37 PM Gilles Gouaillardet <
gilles.gouaillar...@gmail.com> wrote:

> Let me take a step back...
>
> Looking at the code, it should work ... except it does not :-(
>
>
> $ configure --enable-mca-no-build=io-romio341
> [...]
> checking which components should be disabled... io-romio341
> [...]
> +++ Configuring MCA framework io
> checking for no configure components in framework io...
> checking for m4 configure components in framework io... ompio, romio341
>
> --- MCA component io:ompio (m4 configuration macro)
> checking for MCA component io:ompio compile mode... static
> checking if MCA component io:ompio can compile... yes
>
> --- MCA component io:romio341 (m4 configuration macro)
> checking for MCA component io:romio341 compile mode... static
> checking if want ROMIO component... yes
> checking if MPI profiling is enabled... yes
>
> *** Configuring ROMIO distribution
> configure: OPAL configuring in 3rd-party/romio341
>
> On Fri, Nov 18, 2022 at 5:18 PM Gilles Gouaillardet <
> gilles.gouaillar...@gmail.com> wrote:
>
>> Well, --enable-mca-no-build=io-romio341 works strictly speaking (it does
>> not build the io/romio341 component).
>>
>> That being said, it does not prevent 3rd-party/romio from being built,
>> and this is what fails with gcc 4.8
>>
>> I will file an issue in order to keep track of that.
>>
>>
>> Cheers,
>>
>> Gilles
>>
>> On Fri, Nov 18, 2022 at 9:07 AM Barrett, Brian via devel <
>> devel@lists.open-mpi.org> wrote:
>>
>>> --enable-mca-no-build=io-romio341 should still work.  Or just
>>> --disable-io-romio.
>>>
>>> No comment around the RHEL7 part; that's pretty old, but I don't think
>>> we've officially said it is too old.  Probably something worth filing a
>>> ticket for so that we can run to ground before 5.0 release.  Oddly, CI
>>> continues to pass on RHEL7 in AWS.  I'm not sure what we've done to cause
>>> that, but also worth investigating.
>>>
>>> Brian
>>>
>>> On 11/14/22, 10:32 PM, "devel on behalf of Gilles Gouaillardet via
>>> devel" >> devel@lists.open-mpi.org> wrote:
>>>
>>> CAUTION: This email originated from outside of the organization. Do
>>> not click links or open attachments unless you can confirm the sender and
>>> know the content is safe.
>>>
>>>
>>>
>>> Folks,
>>>
>>>
>>> I tried to build on a RHEL7 like, and it fails at make time because
>>> ROMIO requires stdatomic.h
>>>
>>> (it seems this file is only available from GCC 4.9)
>>>
>>>
>>> Are we supposed to be able to build Open MPI 5 with GCC 4.8 (e.g.
>>> stock
>>> RHEL7 compiler)?
>>>
>>>
>>> --enable-mca-no-build=io-romio314 cannot help since ROMIO has been
>>> moved
>>> to 3rd-party
>>>
>>> is there an other workaround? if not, should be add something like
>>> --enable-3rd-party-no-build=romio?
>>>
>>>
>>> Cheers,
>>>
>>>
>>> Gilles
>>>
>>> On 10/22/2022 6:23 AM, Austen W Lauria via devel wrote:
>>> > Open MPI v5.0.0rc9 is now available for testing
>>> > athttps://www.open-mpi.org/software/ompi/v5.0/
>>> > <https://www.open-mpi.org/software/ompi/v5.0/>.
>>> >
>>> > Please test, and send feedback either via the user mailing lists,
>>> or
>>> > create an issue athttps://github.com/open-mpi/ompi/issues/
>>> > <https://github.com/open-mpi/ompi/issues/> .
>>> >
>>> > Seehttps://docs.open-mpi.org/en/v5.0.x/news/news-v5.0.x.html
>>> > <https://docs.open-mpi.org/en/v5.0.x/news/news-v5.0.x.html> for a
>>> list
>>> > of changes since rc8.
>>> >
>>> > Thank you,
>>> > v5.0 Release Managers
>>>
>>>
>>>

Re: [OMPI devel] Open MPI v5.0.0rc9 is available for testing

[Gilles Gouaillardet via devel]

Let me take a step back...

Looking at the code, it should work ... except it does not :-(


$ configure --enable-mca-no-build=io-romio341
[...]
checking which components should be disabled... io-romio341
[...]
+++ Configuring MCA framework io
checking for no configure components in framework io...
checking for m4 configure components in framework io... ompio, romio341

--- MCA component io:ompio (m4 configuration macro)
checking for MCA component io:ompio compile mode... static
checking if MCA component io:ompio can compile... yes

--- MCA component io:romio341 (m4 configuration macro)
checking for MCA component io:romio341 compile mode... static
checking if want ROMIO component... yes
checking if MPI profiling is enabled... yes

*** Configuring ROMIO distribution
configure: OPAL configuring in 3rd-party/romio341

On Fri, Nov 18, 2022 at 5:18 PM Gilles Gouaillardet <
gilles.gouaillar...@gmail.com> wrote:

> Well, --enable-mca-no-build=io-romio341 works strictly speaking (it does
> not build the io/romio341 component).
>
> That being said, it does not prevent 3rd-party/romio from being built, and
> this is what fails with gcc 4.8
>
> I will file an issue in order to keep track of that.
>
>
> Cheers,
>
> Gilles
>
> On Fri, Nov 18, 2022 at 9:07 AM Barrett, Brian via devel <
> devel@lists.open-mpi.org> wrote:
>
>> --enable-mca-no-build=io-romio341 should still work.  Or just
>> --disable-io-romio.
>>
>> No comment around the RHEL7 part; that's pretty old, but I don't think
>> we've officially said it is too old.  Probably something worth filing a
>> ticket for so that we can run to ground before 5.0 release.  Oddly, CI
>> continues to pass on RHEL7 in AWS.  I'm not sure what we've done to cause
>> that, but also worth investigating.
>>
>> Brian
>>
>> On 11/14/22, 10:32 PM, "devel on behalf of Gilles Gouaillardet via
>> devel" > devel@lists.open-mpi.org> wrote:
>>
>> CAUTION: This email originated from outside of the organization. Do
>> not click links or open attachments unless you can confirm the sender and
>> know the content is safe.
>>
>>
>>
>> Folks,
>>
>>
>> I tried to build on a RHEL7 like, and it fails at make time because
>> ROMIO requires stdatomic.h
>>
>> (it seems this file is only available from GCC 4.9)
>>
>>
>> Are we supposed to be able to build Open MPI 5 with GCC 4.8 (e.g.
>> stock
>> RHEL7 compiler)?
>>
>>
>> --enable-mca-no-build=io-romio314 cannot help since ROMIO has been
>> moved
>> to 3rd-party
>>
>> is there an other workaround? if not, should be add something like
>> --enable-3rd-party-no-build=romio?
>>
>>
>> Cheers,
>>
>>
>> Gilles
>>
>> On 10/22/2022 6:23 AM, Austen W Lauria via devel wrote:
>> > Open MPI v5.0.0rc9 is now available for testing
>> > athttps://www.open-mpi.org/software/ompi/v5.0/
>> > <https://www.open-mpi.org/software/ompi/v5.0/>.
>> >
>> > Please test, and send feedback either via the user mailing lists, or
>> > create an issue athttps://github.com/open-mpi/ompi/issues/
>> > <https://github.com/open-mpi/ompi/issues/> .
>> >
>> > Seehttps://docs.open-mpi.org/en/v5.0.x/news/news-v5.0.x.html
>> > <https://docs.open-mpi.org/en/v5.0.x/news/news-v5.0.x.html> for a
>> list
>> > of changes since rc8.
>> >
>> > Thank you,
>> > v5.0 Release Managers
>>
>>
>>

Re: [OMPI devel] Open MPI v5.0.0rc9 is available for testing

[Gilles Gouaillardet via devel]

Folks,


I tried to build on a RHEL7 like, and it fails at make time because 
ROMIO requires stdatomic.h


(it seems this file is only available from GCC 4.9)


Are we supposed to be able to build Open MPI 5 with GCC 4.8 (e.g. stock 
RHEL7 compiler)?



--enable-mca-no-build=io-romio314 cannot help since ROMIO has been moved 
to 3rd-party


is there an other workaround? if not, should be add something like 
--enable-3rd-party-no-build=romio?



Cheers,


Gilles

On 10/22/2022 6:23 AM, Austen W Lauria via devel wrote:
Open MPI v5.0.0rc9 is now available for testing 
athttps://www.open-mpi.org/software/ompi/v5.0/ 
.


Please test, and send feedback either via the user mailing lists, or 
create an issue athttps://github.com/open-mpi/ompi/issues/ 
 .


Seehttps://docs.open-mpi.org/en/v5.0.x/news/news-v5.0.x.html 
 for a list 
of changes since rc8.


Thank you,
v5.0 Release Managers

Re: [OMPI devel] Proposed change to Cuda dependencies in Open MPI

[Gilles Gouaillardet via devel]

William,

What is the desired behavior if Open MPI built with CUDA is used on a
system where CUDA is not available or cannot be used because of ABI
compatibility issues?
 - issue a warning (could not open the DSO because of unsatisfied
dependencies)?
 - silently ignore the CUDA related components?

I guess this should be configurable by yet an other MCA parameter, but that
begs the question of what should be the default value for this parameter.


Cheers,

Gilles


On Sat, Sep 10, 2022 at 6:25 AM Zhang, William via devel <
devel@lists.open-mpi.org> wrote:

> Hello interested parties,
>
>
>
> As part of the work for the accelerator framework, the non standard
> behavior of the existing cuda code in Open MPI is being reworked. One of
> the proposed changes involves a change to the behavior of linking/compiling
> cuda components.
>
>
>
> Currently, cuda functions are loaded dynamically using dlopen and stored
> in a function pointer table, with some code to search through typical paths
> to locate libcuda. This means that we can compile Open MPI
> –with-cuda=/path/to/cuda and the resulting build should work on both cuda
> and non cuda environments.
>
>
>
> The change we are making involves removing the function pointer table and
> instead, having relevant components have a direct dependency on libcuda.
> This is in line with the rest of Open MPI’s MCA system where you can build
> components as dsos.
>
>
>
> The difference here are: Open MPI will call libcuda functions directly and
> components that have a cuda dependency will be built as dso’s (ie.
> –with-cuda=/path/to/cuda/ –enable-mca-dso=accelerator-cuda). During
> linking, these dso’s may fail to load, such as on a non cuda environment,
> but this won’t prevent Open MPI from functioning. A related work -
> https://github.com/open-mpi/ompi/pull/10763 - to have an option to
> silence warnings that occur in this expected behavior path is also being
> worked on.
>
>
> From a user behavior, nothing changes. From compilation, dependent
> components will need to be built as dso’s. From code, we can remove dlopen
> dependency for cuda builds, standardize the cuda code with the rest of Open
> MPI, and remove code involved with storing function pointers and detecting
> libcuda location.
>
>
>
> Please provide feedback if you have any suggestions or are against these
> changes.
>
>
>
> Thanks,
>
> William Zhang
>

Re: [OMPI devel] Open MPI Java MPI bindings

[Gilles Gouaillardet via devel]

Hi,

Same impression here.
There are bug reports once in a while (both Open MPI mailing list and Stack
Overflow).
I remember one used the Java bindings to teach MPI.

So my gut feeling is the number of active users is small but not zero.


Cheers,

Gilles

On Wed, Aug 10, 2022 at 8:48 PM t-kawashima--- via devel <
devel@lists.open-mpi.org> wrote:

> Hi,
>
> Fujitsu MPI, which is currently based on Open MPI v4.0.x, supports Java
> bindings.
>
> It is because one of our customers requested to support it decade or so
> ago. I don't know users are still using it. Fujitsu MPI development team
> regularly receives trouble/request reports but does not receive Java ones
> recently. So users are few, if any.
>
> We Fujitsu don't mind dropping Java binding in Open MPI v5.0.x.
>
> Thanks,
> Takahiro Kawashima,
> Fujitsu
>
> > During a planning meeting for Open MPI v5.0.0 today, the question came
> up: is anyone using the Open MPI Java bindings?
> >
> > These bindings are not official MPI Forum bindings -- they are an Open
> MPI-specific extension.  They were added a few years ago as a result of a
> research project.
> >
> > We ask this question because we're wondering if it's worthwhile to bring
> these bindings forward to the v5.0.x series, or whether we should remove
> them from v5.0.x, and just leave them available back in the v4.0.x and
> v4.1.x series.
> >
> > Please reply here to this list if you are using the Open MPI Java
> bindings, or know of anyone who is using them.
> >
> > Thank you!

Re: [OMPI devel] Possible Bug / Invalid Read in Ialltoallw

[Gilles Gouaillardet via devel]

Thanks again Damian,

I think the root cause is we call mca_topo_base_neighbor_count() instead of
ompi_comm_size() here.
It seems the implicit assumption is one would call
MPI_Ineighbor_alltoallw() on a cartesian communicator ... which is
obviously wrong: it is legit to call MPI_Ialltoallw(), even if the
communicator is cartesian.

You can try the attached patch (for the v4.1 series).

Note this patch breaks the internal ABI, that is why we will likely end up
with a different one for the v4.1 series.


Cheers,

Gilles

On Thu, May 5, 2022 at 1:10 AM Damian Marek via devel <
devel@lists.open-mpi.org> wrote:

> Hello,
>
> I made an example that triggers the issue. I had to get a little creative
> with how to trigger the crash, since it does not appear if the memory
> allocated for the send and recv types happens to be set to 0 (although
> valgrind still reports an invalid read).
>
> Communicator is an intra-communicator with a cartesian topology. It takes
> the first branch into "mca_topo_base_neighbor_count" and sets the wrong
> rcount and scount, at least to my understanding.
>
> Regards,
> Damian
> --
> *From:* devel  on behalf of George
> Bosilca via devel 
> *Sent:* Wednesday, May 4, 2022 10:20 AM
> *To:* Open MPI Developers 
> *Cc:* George Bosilca 
> *Subject:* Re: [OMPI devel] Possible Bug / Invalid Read in Ialltoallw
>
> Damien,
>
> As Gilles indicated an example would be great. Meanwhile, as you already
> have access to the root cause with a debugger, can you check what branch of
> the if regarding the communicator type in the
> ompi_coll_base_retain_datatypes_w function is taken. What is the
> communicator type ? Intra or inter ? with or without topology ?
>
> Thanks,
>   George.
>
>
> On Wed, May 4, 2022 at 9:35 AM Gilles Gouaillardet via devel <
> devel@lists.open-mpi.org> wrote:
>
> Damian,
>
> Thanks for the report!
>
> could you please trim your program and share it so I can have a look?
>
>
> Cheers,
>
> Gilles
>
>
> On Wed, May 4, 2022 at 10:27 PM Damian Marek via devel <
> devel@lists.open-mpi.org> wrote:
>
> Hello,
>
> I have been getting intermittent memory corruptions and segmentation
> faults while using Ialltoallw in OpenMPI v4.0.3. Valgrind also reports an
> invalid read in the "ompi_coll_base_retain_datatypes_w" function defined in
> "coll_base_util.c".
>
> Running with a debug build of ompi an assertion fails as well:
>
> base/coll_base_util.c:274: ompi_coll_base_retain_datatypes_w: Assertion
> `OPAL_OBJ_MAGIC_ID == ((opal_object_t *) (stypes[i]))->obj_magic_id' failed.
>
> I think it is related to the fact that I am using a communicator created
> with 2D MPI_Cart_create followed by getting 2 subcommunicators from
> MPI_Cart_sub, in some cases one of the dimensions is 1. In
> "ompi_coll_base_retain_datatypes_w" the neighbour count is used to find
> "rcount" and "scount" at line 267. In my bug case it returns 2 for both,
> but I believe it should be 1 since that is the comm size and the amount of
> memory I have allocated for sendtypes and recvtypes. Then, an invalid read
> happens at 274 and 280.
>
> Regards,
> Damian
>
>
>
>
>
>
>


alltoallw.patch
Description: Binary data

Re: [OMPI devel] Possible Bug / Invalid Read in Ialltoallw

[Gilles Gouaillardet via devel]

Damian,

Thanks for the report!

could you please trim your program and share it so I can have a look?


Cheers,

Gilles


On Wed, May 4, 2022 at 10:27 PM Damian Marek via devel <
devel@lists.open-mpi.org> wrote:

> Hello,
>
> I have been getting intermittent memory corruptions and segmentation
> faults while using Ialltoallw in OpenMPI v4.0.3. Valgrind also reports an
> invalid read in the "ompi_coll_base_retain_datatypes_w" function defined in
> "coll_base_util.c".
>
> Running with a debug build of ompi an assertion fails as well:
>
> base/coll_base_util.c:274: ompi_coll_base_retain_datatypes_w: Assertion
> `OPAL_OBJ_MAGIC_ID == ((opal_object_t *) (stypes[i]))->obj_magic_id' failed.
>
> I think it is related to the fact that I am using a communicator created
> with 2D MPI_Cart_create followed by getting 2 subcommunicators from
> MPI_Cart_sub, in some cases one of the dimensions is 1. In
> "ompi_coll_base_retain_datatypes_w" the neighbour count is used to find
> "rcount" and "scount" at line 267. In my bug case it returns 2 for both,
> but I believe it should be 1 since that is the comm size and the amount of
> memory I have allocated for sendtypes and recvtypes. Then, an invalid read
> happens at 274 and 280.
>
> Regards,
> Damian
>
>
>
>
>
>
>

Re: [OMPI devel] Script-based wrapper compilers

[Gilles Gouaillardet via devel]

Jeff,

Cross compilation is the recommended way on Fugaku.
In all fairness, even if Fujitsu MPI is based on Open MPI, they built the
MPI wrappers (that invoke the cross compilers) on top of opal (read not the
scripts).

Cheers,

Gilles

On Fri, Mar 25, 2022 at 1:06 AM Jeff Squyres (jsquyres) 
wrote:

> Gilles --
>
> Do you know if anyone is actually cross compiling?  I agree that this is
> in the "nice to have" category, but it is costing Brian time -- if no one
> is using this functionality, it's not worth the time.  If people are using
> this functionality, then it's potentially worth the time.
>
> --
> Jeff Squyres
> jsquy...@cisco.com
>
> ________
> From: devel  on behalf of Gilles
> Gouaillardet via devel 
> Sent: Wednesday, March 23, 2022 10:28 PM
> To: Open MPI Developers
> Cc: Gilles Gouaillardet
> Subject: Re: [OMPI devel] Script-based wrapper compilers
>
> Brian,
>
> My 0.02 US$
>
> Script based wrapper compilers are very useful when cross compiling,
> so ideally, they should be maintained.
>
> Cheers,
>
> Gilles
>
> On Thu, Mar 24, 2022 at 11:18 AM Barrett, Brian via devel <
> devel@lists.open-mpi.org<mailto:devel@lists.open-mpi.org>> wrote:
> Does anyone still use the script based wrapper compilers?  I have been
> working on fixing a number of static library compile issues caused by us
> historically not having been great about tracking library dependencies and
> the OMPI/PMIx/PRRTE split.  Part of this is some fairly significant
> modifications to the wrapper compilers (here's the PMIx version:
> https://github.com/openpmix/openpmix/commit/e15de4b52f2d331297bbca31beb54b5a377557bc).
> It would be easiest to just remove the script based wrapper compilers, but
> I'll update them if someone uses them.
>
> Thanks,
>
> Brian
>
>

Re: [OMPI devel] Script-based wrapper compilers

[Gilles Gouaillardet via devel]

Brian,

My 0.02 US$

Script based wrapper compilers are very useful when cross compiling,
so ideally, they should be maintained.

Cheers,

Gilles

On Thu, Mar 24, 2022 at 11:18 AM Barrett, Brian via devel <
devel@lists.open-mpi.org> wrote:

> Does anyone still use the script based wrapper compilers?  I have been
> working on fixing a number of static library compile issues caused by us
> historically not having been great about tracking library dependencies and
> the OMPI/PMIx/PRRTE split.  Part of this is some fairly significant
> modifications to the wrapper compilers (here's the PMIx version:
> https://github.com/openpmix/openpmix/commit/e15de4b52f2d331297bbca31beb54b5a377557bc).
> It would be easiest to just remove the script based wrapper compilers, but
> I'll update them if someone uses them.
>
> Thanks,
>
> Brian
>
>

Re: [OMPI devel] OpenMPI and maker - Multiple messages

[Gilles Gouaillardet via devel]

Thomas,

in order to test Open MPI, you can download
https://raw.githubusercontent.com/open-mpi/ompi/master/examples/hello_c.c

and build it

$ mpicc -o hello_c examples/hello_c.c

and then run
$ mpiexec --mca btl ^openib -N 5 ./hello_c

It should display 5 different lines (rank 0 to 4 of 5).
If you see the same line (I am 0 of 1) 5 times, this indicates an issue.

Cheers,

Gilles

On Thu, Feb 11, 2021 at 12:36 AM Thomas Eylenbosch via devel <
devel@lists.open-mpi.org> wrote:

> Hello
>
>
>
> We are trying to run maker(http://www.yandell-lab.org/software/maker.html)
> in combination with OpenMPI
>
>
>
> But when we are trying to submit a job with the maker and openmpi,
>
> We see the following error in the log file:
>
> --Next Contig--
>
> #-
>
> Another instance of maker is processing *this* contig!!
>
> SeqID: chrA10
>
> Length: 17398227
>
> #-
>
>
>
> According to
>
>
> http://gmod.827538.n3.nabble.com/Does-maker-support-muti-processing-for-a-single-long-fasta-sequence-using-openMPI-td4061342.html
>
>
>
> We have to run the following command  mpiexec -mca btl ^openib -n 40
> maker -help
>
> “If you get a single help message then everything is fine.  If you get 40
> help messages, then MPI is not communicating correctly.”
>
>
>
> We are using the following command to demonstrate what is going wrong:
>
> mpiexec -mca btl ^openib -N 5 gcc --version
>
> gcc (GCC) 10.2.0
>
> Copyright (C) 2020 Free Software Foundation, Inc.
>
> This is free software; see the source for copying conditions.  There is NO
>
> warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
>
>
>
> gcc (GCC) 10.2.0
>
> Copyright (C) 2020 Free Software Foundation, Inc.
>
> This is free software; see the source for copying conditions.  There is NO
>
> warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
>
>
>
> gcc (GCC) 10.2.0
>
> Copyright (C) 2020 Free Software Foundation, Inc.
>
> This is free software; see the source for copying conditions.  There is NO
>
> warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
>
>
>
> gcc (GCC) 10.2.0
>
> Copyright (C) 2020 Free Software Foundation, Inc.
>
> This is free software; see the source for copying conditions.  There is NO
>
> warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
>
>
>
> gcc (GCC) 10.2.0
>
> Copyright (C) 2020 Free Software Foundation, Inc.
>
> This is free software; see the source for copying conditions.  There is NO
>
> warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
>
>
>
>
>
>
>
> So we are getting the message 5 times, which means OpenMPI is not
> correctly installed on our cluster?
>
> We are using EasyBuild to build/install our OpenMPI module. ( with the
> default OpenMPI easyblock module)
>
>
>
> Best regards / met vriendelijke groeten
> * Thomas Eylenbosch*
> DevOps Engineer (OnSite), Gluo N.V.
>
>
>
> *Currently available at BASF Belgium Coordination Center CommV*
> Email: thomas.eylenbo...@partners.basf.com
> Postal Address: BASF Belgium Coordination Center CommV, Technologiepark
> 101, 9052 Gent Zwijnaarde, Belgium
>
>
>
> BASF Belgium Coordination Center CommV
>
> Scheldelaan 600, 2040 Antwerpen, België
>
> RPR Antwerpen (afd. Antwerpen)
>
> BTW BE0862.390.376
>
>
>
> *www.basf.be *
>
>
>
> Deutsche Bank AG
>
> IBAN: BE43 8262 8044 4801
>
> BIC: DEUTBEBE
>
>
>
>
>
> Information on data protection can be found here:
> https://www.basf.com/global/en/legal/data-protection-at-basf.html
>
>
>
>
>
>
>
>
>

Re: [OMPI devel] configure problem on master

[Gilles Gouaillardet via devel]

Edgar and Jeff,

the root cause is we do not use pthreads here any more:

>From master config.log

configure:226861: gcc -std=gnu99 -o conftest -O3 -DNDEBUG
-finline-functions -fno-strict-aliasing -mcx16
-I/home/usersup/gilles/build/ompi-master/3rd-party/openpmix/include
-I/home/usersup/gilles/src/ompi-master/3rd-party/openpmix/include
-I/usr/local/include -I/usr/local/include  conftest.c -lrt -lm -lutil
>&5

/usr/bin/ld: /tmp/ccDiuQBc.o: undefined reference to symbol
'sem_open@@GLIBC_2.2.5'

//usr/lib64/libpthread.so.0: error adding symbols: DSO missing from command line

collect2: error: ld returned 1 exit status


The test passes on v4.1.x because the '-pthread' flag is passed

Cheers,

Gilles

On Sat, Feb 6, 2021 at 11:27 AM Gabriel, Edgar via devel
 wrote:
>
> The library that contains sem_init and sem_open, the two functions that we 
> check for in the configure script
>
>
>
> From: Jeff Squyres (jsquyres) 
> Sent: Friday, February 5, 2021 7:31 PM
> To: Open MPI Developers List 
> Cc: Gabriel, Edgar 
> Subject: Re: [OMPI devel] configure problem on master
>
>
>
> Which library?
>
>
>
> On Feb 5, 2021, at 4:46 PM, Gabriel, Edgar via devel 
>  wrote:
>
>
>
> I just noticed another ‘new’ aspect of configure on master. The sharedfp/sm 
> component is now not compiled anymore, I double checked that it is still 
> correctly detected and handled on 4.1, so it must be a recent change.
>
>
>
> On master:
>
> --snip--
>
>
>
> --- MCA component sharedfp:sm (m4 configuration macro)
>
> checking for MCA component sharedfp:sm compile mode... dso
>
> checking semaphore.h usability... yes
>
> checking semaphore.h presence... yes
>
> checking for semaphore.h... yes
>
> checking for sem_open... no
>
> checking for semaphore.h... (cached) yes
>
> checking for sem_init... no
>
> checking if MCA component sharedfp:sm can compile... no
>
>
>
> --snip --
>
>
>
> while e.g. on 4.1 on exactly the same platform:
>
>
>
> --snip--
>
> --- MCA component sharedfp:sm (m4 configuration macro)
>
> checking for MCA component sharedfp:sm compile mode... dso
>
> checking semaphore.h usability... yes
>
> checking semaphore.h presence... yes
>
> checking for semaphore.h... yes
>
> checking for sem_open... yes
>
> checking for semaphore.h... (cached) yes
>
> checking for sem_init... yes
>
> checking if MCA component sharedfp:sm can compile... yes
>
> ---snip---
>
>
>
> it looks like a library that is required for the semaphore operations was be 
> default included previously, and is not anymore?
>
> Thanks
>
> Edgar
>
>
>
> From: devel  On Behalf Of Gabriel, Edgar 
> via devel
> Sent: Thursday, February 4, 2021 2:15 PM
> To: Open MPI Developers 
> Cc: Gabriel, Edgar 
> Subject: Re: [OMPI devel] configure problem on master
>
>
>
> excellent, thanks! I have meanwhile a more detailed suspicion:
>
> --
>
> looking for library without search path
>
> checking for library containing llapi_file_create... no
>
> looking for library in /opt/lustre/2.12.2/lib64
>
> checking for library containing llapi_file_create... (cached) no
>
> --
>
>
>
> the liblustrapi library is in the /opt/lustre/2.12.2/lib64, so the configure 
> script should not be using the cached value (no) but really check again. This 
> seems to be the key difference between the ompi and openpmix scripts.
>
>
>
> (for comparison the ompi output is
>
>
>
> looking for library in lib
>
> checking for library containing llapi_file_create... no
>
> looking for library in lib64
>
> checking for library containing llapi_file_create... -llustreapi )
>
>
>
>
>
> Thanks
>
> Edgar
>
>
>
> From: devel  On Behalf Of Ralph Castain via 
> devel
> Sent: Thursday, February 4, 2021 2:02 PM
> To: OpenMPI Devel 
> Cc: Ralph Castain 
> Subject: Re: [OMPI devel] configure problem on master
>
>
>
> Sounds like I need to resync the PMIx lustre configury with the OMPI one - 
> I'll do that.
>
>
>
>
>
> On Feb 4, 2021, at 11:56 AM, Gabriel, Edgar via devel 
>  wrote:
>
>
>
> I have a weird problem running configure on master on our cluster. Basically, 
> configure fails when I request lustre support, but not from ompio but 
> openpmix.
>
>
>
> What makes our cluster setup maybe a bit special is that the lustre libraries 
> are not installed in the standard path, but in /opt, and thus we provide the 
> --with-lustre=/opt/lustre/2.12.2 as an option.
>
> If I remove the 3rd-party/openpmix/src/mca/pstrg/lustre component, the 
> configure script finishes correctly.
>
>
>
> I looked at the ompi vs. openmpix check_lustre configure scripts, I could not 
> detect on a quick glance any difference that would explain why the script is 
> failing in one instance but not the other, but the openpmix version does seem 
> to go through some additional hoops (checking separately for the include 
> directory, the lib and lib64 directories etc), so it might be a difference  
> in the PMIX_ macros vs. the OPAL_ macros.
>
>
>
> --snip--
>
>
>
> --- MCA component fs:lustre (m4 configuration macro)
>
> checking for MCA component 

Re: [OMPI devel] Segmentation fault in MPI_Get in the Vader/sm module

[Gilles Gouaillardet via devel]

Max,

Thanks for the report and suggesting a solution!

Could you please issue a pull request so your fix can be reviewed?
note the commit has to be signed-off (IANAL, make sure you understand
the implications)
in order to be accepted upstream.

Cheers,

Gilles

On Thu, Feb 4, 2021 at 6:40 AM Max Schrötter via devel
 wrote:
>
> Hi everyone,
>
> I recently opened an issue on github, before reading on the OpenMPI site
> that I should have posted my problem here, sorry.
>
> I get a segmentation fault when using one sided communication on a
> new communicator created by MPI_Comm_split.
>
> All the details are on github : https://github.com/open-mpi/ompi/issues/8434#
>
> I also might have found a possible solution.
>
> It would be great if someone could have a look at the issue and try to
> reproduce it or tell me if my test code is wrong.
>
> Thanks,
> Max

Re: [OMPI devel] mpirun 4.1.0 segmentation fault

[Gilles Gouaillardet via devel]

Andrej,

that really looks like a SLURM issue that does not involve Open MPI

In order to confirm, you can

$ salloc -N 2 -n 2
/* and then from the allocation */
srun hostname

If this does not work, then this is a SLURM issue you have to fix.
Once fixed, I am confident Open MPI will just work

Cheers,

Gilles

On Tue, Feb 2, 2021 at 10:22 AM Andrej Prsa via devel
 wrote:
>
> Hi Gilles,
>
> > Here is what you can try
> >
> > $ salloc -N 4 -n 384
> > /* and then from the allocation */
> >
> > $ srun -n 1 orted
> > /* that should fail, but the error message can be helpful */
> >
> > $ /usr/local/bin/mpirun --mca plm slurm --mca plm_base_verbose 10 true
>
> andrej@terra:~/system/tests/MPI$ salloc -N 4 -n 384
> salloc: Granted job allocation 837
> andrej@terra:~/system/tests/MPI$ srun -n 1 orted
> srun: Warning: can't run 1 processes on 4 nodes, setting nnodes to 1
> srun: launch/slurm: launch_p_step_launch: StepId=837.0 aborted before
> step completely launched.
> srun: Job step aborted: Waiting up to 32 seconds for job step to finish.
> srun: error: task 0 launch failed: Unspecified error
> andrej@terra:~/system/tests/MPI$ /usr/local/bin/mpirun -mca plm slurm
> -mca plm_base_verbose 10 true
> [terra:179991] mca: base: components_register: registering framework plm
> components
> [terra:179991] mca: base: components_register: found loaded component slurm
> [terra:179991] mca: base: components_register: component slurm register
> function successful
> [terra:179991] mca: base: components_open: opening plm components
> [terra:179991] mca: base: components_open: found loaded component slurm
> [terra:179991] mca: base: components_open: component slurm open function
> successful
> [terra:179991] mca:base:select: Auto-selecting plm components
> [terra:179991] mca:base:select:(  plm) Querying component [slurm]
> [terra:179991] [[INVALID],INVALID] plm:slurm: available for selection
> [terra:179991] mca:base:select:(  plm) Query of component [slurm] set
> priority to 75
> [terra:179991] mca:base:select:(  plm) Selected component [slurm]
> [terra:179991] plm:base:set_hnp_name: initial bias 179991 nodename hash
> 2928217987
> [terra:179991] plm:base:set_hnp_name: final jobfam 7711
> [terra:179991] [[7711,0],0] plm:base:receive start comm
> [terra:179991] [[7711,0],0] plm:base:setup_job
> [terra:179991] [[7711,0],0] plm:slurm: LAUNCH DAEMONS CALLED
> [terra:179991] [[7711,0],0] plm:base:setup_vm
> [terra:179991] [[7711,0],0] plm:base:setup_vm creating map
> [terra:179991] [[7711,0],0] plm:base:setup_vm add new daemon [[7711,0],1]
> [terra:179991] [[7711,0],0] plm:base:setup_vm assigning new daemon
> [[7711,0],1] to node node9
> [terra:179991] [[7711,0],0] plm:base:setup_vm add new daemon [[7711,0],2]
> [terra:179991] [[7711,0],0] plm:base:setup_vm assigning new daemon
> [[7711,0],2] to node node10
> [terra:179991] [[7711,0],0] plm:base:setup_vm add new daemon [[7711,0],3]
> [terra:179991] [[7711,0],0] plm:base:setup_vm assigning new daemon
> [[7711,0],3] to node node11
> [terra:179991] [[7711,0],0] plm:base:setup_vm add new daemon [[7711,0],4]
> [terra:179991] [[7711,0],0] plm:base:setup_vm assigning new daemon
> [[7711,0],4] to node node12
> [terra:179991] [[7711,0],0] plm:slurm: launching on nodes
> node9,node10,node11,node12
> [terra:179991] [[7711,0],0] plm:slurm: Set prefix:/usr/local
> [terra:179991] [[7711,0],0] plm:slurm: final top-level argv:
>  srun --ntasks-per-node=1 --kill-on-bad-exit --ntasks=4 orted -mca
> ess "slurm" -mca ess_base_jobid "505348096" -mca ess_base_vpid "1" -mca
> ess_base_num_procs "5" -mca orte_node_regex
> "terra,node[1:9],node[2:10-12]@0(5)" -mca orte_hnp_uri
> "505348096.0;tcp://10.9.2.10,192.168.1.1:38995" -mca plm_base_verbose "10"
> [terra:179991] [[7711,0],0] plm:slurm: reset PATH:
> /usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
> [terra:179991] [[7711,0],0] plm:slurm: reset LD_LIBRARY_PATH: /usr/local/lib
> srun: launch/slurm: launch_p_step_launch: StepId=837.1 aborted before
> step completely launched.
> srun: Job step aborted: Waiting up to 32 seconds for job step to finish.
> srun: error: task 3 launch failed: Unspecified error
> srun: error: task 1 launch failed: Unspecified error
> srun: error: task 2 launch failed: Unspecified error
> srun: error: task 0 launch failed: Unspecified error
> [terra:179991] [[7711,0],0] plm:slurm: primary daemons complete!
> [terra:179991] [[7711,0],0] plm:base:receive stop comm
> [terra:179991] mca: base: close: component slurm closed
> [terra:179991] mca: base: close: unloading component slurm
>
> This is what I'm seeing in slurmctld.log:
>
> [2021-02-01T20:15:18.358] sched: _slurm_rpc_allocate_resources JobId=837
> NodeList=node[9-12] usec=537
> [2021-02-01T20:15:26.815] error: mpi_hook_slurmstepd_prefork failure for
> 0x557ce5b92960s on node9
> [2021-02-01T20:15:59.621] error: mpi_hook_slurmstepd_prefork failure for
> 0x55cc6c89a7e0s on node12
> [2021-02-01T20:15:59.621] 

Re: [OMPI devel] mpirun 4.1.0 segmentation fault

[Gilles Gouaillardet via devel]

e_node_regex
> "terra,node[1:9],node[2:10-12]@0(5)" -mca orte_hnp_uri
> "360382464.0;tcp://10.9.2.10,192.168.1.1:45597" -mca plm_base_verbose
> "10" -mca plt "slurm"
> srun: launch/slurm: launch_p_step_launch: StepId=836.0 aborted before
> step completely launched.
> srun: Job step aborted: Waiting up to 32 seconds for job step to finish.
> srun: error: task 0 launch failed: Unspecified error
> srun: error: task 3 launch failed: Unspecified error
> srun: error: task 2 launch failed: Unspecified error
> srun: error: task 1 launch failed: Unspecified error
> [terra:177267] [[5499,0],0] plm:slurm: primary daemons complete!
> [terra:177267] [[5499,0],0] plm:base:receive stop comm
> [terra:177267] mca: base: close: component slurm closed
> [terra:177267] mca: base: close: unloading component slurm
>
> Thanks, as always,
> Andrej
>
>
> On 2/1/21 7:50 PM, Gilles Gouaillardet via devel wrote:
> > Andrej,
> >
> > you *have* to invoke
> > mpirun --mca plm slurm ...
> > from a SLURM allocation, and SLURM_* environment variables should have
> > been set by SLURM
> > (otherwise, this is a SLURM error out of the scope of Open MPI).
> >
> > Here is what you can try (and send the logs if that fails)
> >
> > $ salloc -N 4 -n 384
> > and once you get the allocation
> > $ env | grep ^SLURM_
> > $ mpirun --mca plm_base_verbose 10 --mca plm slurm true
> >
> >
> > Cheers,
> >
> > Gilles
> >
> > On Tue, Feb 2, 2021 at 9:27 AM Andrej Prsa via devel
> >  wrote:
> >> Hi Gilles,
> >>
> >>> I can reproduce this behavior ... when running outside of a slurm 
> >>> allocation.
> >> I just tried from slurm (sbatch run.sh) and I get the exact same error.
> >>
> >>> What does
> >>> $ env | grep ^SLURM_
> >>> reports?
> >> Empty; no environment variables have been defined.
> >>
> >> Thanks,
> >> Andrej
> >>
>

Re: [OMPI devel] mpirun 4.1.0 segmentation fault

[Gilles Gouaillardet via devel]

Andrej,

you *have* to invoke
mpirun --mca plm slurm ...
from a SLURM allocation, and SLURM_* environment variables should have
been set by SLURM
(otherwise, this is a SLURM error out of the scope of Open MPI).

Here is what you can try (and send the logs if that fails)

$ salloc -N 4 -n 384
and once you get the allocation
$ env | grep ^SLURM_
$ mpirun --mca plm_base_verbose 10 --mca plm slurm true


Cheers,

Gilles

On Tue, Feb 2, 2021 at 9:27 AM Andrej Prsa via devel
 wrote:
>
> Hi Gilles,
>
> > I can reproduce this behavior ... when running outside of a slurm 
> > allocation.
>
> I just tried from slurm (sbatch run.sh) and I get the exact same error.
>
> >
> > What does
> > $ env | grep ^SLURM_
> > reports?
>
> Empty; no environment variables have been defined.
>
> Thanks,
> Andrej
>

Re: [OMPI devel] mpirun 4.1.0 segmentation fault

[Gilles Gouaillardet via devel]

Andrej,

I can reproduce this behavior ... when running outside of a slurm allocation.

What does
$ env | grep ^SLURM_
reports?

Cheers,

Gilles

On Tue, Feb 2, 2021 at 9:06 AM Andrej Prsa via devel
 wrote:
>
> Hi Ralph, Gilles,
>
> > I fail to understand why you continue to think that PMI has anything to do 
> > with this problem. I see no indication of a PMIx-related issue in anything 
> > you have provided to date.
>
> Oh, I went off the traceback that yelled about pmix, and slurm not being
> able to find it until I patched the latest version; I'm an
> astrophysicist pretending to be a sys admin for our research cluster, so
> while I can hold my ground with c, python and technical computing, I'm
> out of my depths when it comes to mpi, pmix, slurm and all that good
> stuff. So I appreciate your patience. I am trying though. :)
>
> > In the output below, it is clear what the problem is - you locked it to the 
> > "slurm" launcher (with -mca plm slurm) and the "slurm" launcher was not 
> > found. Try adding "--mca plm_base_verbose 10" to your cmd line and let's 
> > see why that launcher wasn't accepted.
>
> andrej@terra:~/system/tests/MPI$ mpirun -mca plm_base_verbose 10 -mca
> plm slurm -np 384 -H node15:96,node16:96,node17:96,node18:96 python
> testmpi.py
> [terra:168998] mca: base: components_register: registering framework plm
> components
> [terra:168998] mca: base: components_register: found loaded component slurm
> [terra:168998] mca: base: components_register: component slurm register
> function successful
> [terra:168998] mca: base: components_open: opening plm components
> [terra:168998] mca: base: components_open: found loaded component slurm
> [terra:168998] mca: base: components_open: component slurm open function
> successful
> [terra:168998] mca:base:select: Auto-selecting plm components
> [terra:168998] mca:base:select:(  plm) Querying component [slurm]
> [terra:168998] mca:base:select:(  plm) No component selected!
> --
> It looks like orte_init failed for some reason; your parallel process is
> likely to abort.  There are many reasons that a parallel process can
> fail during orte_init; some of which are due to configuration or
> environment problems.  This failure appears to be an internal failure;
> here's some additional information (which may only be relevant to an
> Open MPI developer):
>
>orte_plm_base_select failed
>--> Returned value Not found (-13) instead of ORTE_SUCCESS
> --
>
> Gilles, I did try all the suggestions from the previous email but that
> led me to think that slurm is the culprit, and now I'm back to openmpi.
>
> Cheers,
> Andrej
>

Re: [OMPI devel] mpirun 4.1.0 segmentation fault

[Gilles Gouaillardet via devel]

Andrej,

My previous email listed other things to try

Cheers,

Gilles

Sent from my iPod

> On Feb 2, 2021, at 6:23, Andrej Prsa via devel  
> wrote:
> 
> The saga continues.
> 
> I managed to build slurm with pmix by first patching slurm using this patch 
> and manually building the plugin:
> 
> https://bugs.schedmd.com/show_bug.cgi?id=10683
> 
> Now srun shows pmix as an option:
> 
> andrej@terra:~/system/tests/MPI$ srun --mpi=list
> srun: MPI types are...
> srun: cray_shasta
> srun: none
> srun: pmi2
> srun: pmix
> srun: pmix_v4
> 
> But when I try to run mpirun with slurm plugin, it still fails:
> 
> andrej@terra:~/system/tests/MPI$ mpirun -mca ess_base_verbose 10 --mca 
> pmix_base_verbose 10 -mca plm slurm -np 384 -H 
> node15:96,node16:96,node17:96,node18:96 python testmpi.py
> [terra:149214] mca: base: components_register: registering framework ess 
> components
> [terra:149214] mca: base: components_register: found loaded component slurm
> [terra:149214] mca: base: components_register: component slurm has no 
> register or open function
> [terra:149214] mca: base: components_register: found loaded component env
> [terra:149214] mca: base: components_register: component env has no register 
> or open function
> [terra:149214] mca: base: components_register: found loaded component pmi
> [terra:149214] mca: base: components_register: component pmi has no register 
> or open function
> [terra:149214] mca: base: components_register: found loaded component tool
> [terra:149214] mca: base: components_register: component tool register 
> function successful
> [terra:149214] mca: base: components_register: found loaded component hnp
> [terra:149214] mca: base: components_register: component hnp has no register 
> or open function
> [terra:149214] mca: base: components_register: found loaded component 
> singleton
> [terra:149214] mca: base: components_register: component singleton register 
> function successful
> [terra:149214] mca: base: components_open: opening ess components
> [terra:149214] mca: base: components_open: found loaded component slurm
> [terra:149214] mca: base: components_open: component slurm open function 
> successful
> [terra:149214] mca: base: components_open: found loaded component env
> [terra:149214] mca: base: components_open: component env open function 
> successful
> [terra:149214] mca: base: components_open: found loaded component pmi
> [terra:149214] mca: base: components_open: component pmi open function 
> successful
> [terra:149214] mca: base: components_open: found loaded component tool
> [terra:149214] mca: base: components_open: component tool open function 
> successful
> [terra:149214] mca: base: components_open: found loaded component hnp
> [terra:149214] mca: base: components_open: component hnp open function 
> successful
> [terra:149214] mca: base: components_open: found loaded component singleton
> [terra:149214] mca: base: components_open: component singleton open function 
> successful
> [terra:149214] mca:base:select: Auto-selecting ess components
> [terra:149214] mca:base:select:(  ess) Querying component [slurm]
> [terra:149214] mca:base:select:(  ess) Querying component [env]
> [terra:149214] mca:base:select:(  ess) Querying component [pmi]
> [terra:149214] mca:base:select:(  ess) Querying component [tool]
> [terra:149214] mca:base:select:(  ess) Querying component [hnp]
> [terra:149214] mca:base:select:(  ess) Query of component [hnp] set priority 
> to 100
> [terra:149214] mca:base:select:(  ess) Querying component [singleton]
> [terra:149214] mca:base:select:(  ess) Selected component [hnp]
> [terra:149214] mca: base: close: component slurm closed
> [terra:149214] mca: base: close: unloading component slurm
> [terra:149214] mca: base: close: component env closed
> [terra:149214] mca: base: close: unloading component env
> [terra:149214] mca: base: close: component pmi closed
> [terra:149214] mca: base: close: unloading component pmi
> [terra:149214] mca: base: close: component tool closed
> [terra:149214] mca: base: close: unloading component tool
> [terra:149214] mca: base: close: component singleton closed
> [terra:149214] mca: base: close: unloading component singleton
> --
> It looks like orte_init failed for some reason; your parallel process is
> likely to abort.  There are many reasons that a parallel process can
> fail during orte_init; some of which are due to configuration or
> environment problems.  This failure appears to be an internal failure;
> here's some additional information (which may only be relevant to an
> Open MPI developer):
> 
>   orte_plm_base_select failed
>   --> Returned value Not found (-13) instead of ORTE_SUCCESS
> --
> 
> I'm at my wits' end what to try, and all ears if anyone has any leads or 
> suggestions.
> 
> Thanks,
> Andrej
> 

Re: [OMPI devel] mpirun 4.1.0 segmentation fault

[Gilles Gouaillardet via devel]

Andrej,

you are now invoking mpirun from a slurm allocation, right?

you can try this:

/usr/local/bin/mpirun -mca plm slurm -np 384 -H
node15:96,node16:96,node17:96,node18:96
python testmpi.py

if it does not work, you can collect more relevant logs with

mpirun -mca plm slurm -mca plm_base_verbose 10 -np 384 -H
node15:96,node16:96,node17:96,node18:96
python testmpi.py

an other test you can do is
srun -N 1 -n 1 orted

that is expected to fail, but it should at least find all its
dependencies and start


Cheers,

Gilles

On Tue, Feb 2, 2021 at 12:32 AM Andrej Prsa via devel
 wrote:
>
> Alright, I rebuilt mpirun and it's working on a local machine. But now
> I'm back to my original problem: running this works:
>
> mpirun -mca plm rsh -np 384 -H node15:96,node16:96,node17:96,node18:96
> python testmpi.py
>
> but running this doesn't:
>
> mpirun -mca plm slurm -np 384 -H node15:96,node16:96,node17:96,node18:96
> python testmpi.py
>
> Here's the verbose output from the latter command:
>
> andrej@terra:~/system/tests/MPI$ mpirun -mca ess_base_verbose 10 --mca
> pmix_base_verbose 10 -mca plm slurm -np 384 -H
> node15:96,node16:96,node17:96,node18:96 python testmpi.py
> [terra:387112] mca: base: components_register: registering framework ess
> components
> [terra:387112] mca: base: components_register: found loaded component slurm
> [terra:387112] mca: base: components_register: component slurm has no
> register or open function
> [terra:387112] mca: base: components_register: found loaded component env
> [terra:387112] mca: base: components_register: component env has no
> register or open function
> [terra:387112] mca: base: components_register: found loaded component pmi
> [terra:387112] mca: base: components_register: component pmi has no
> register or open function
> [terra:387112] mca: base: components_register: found loaded component tool
> [terra:387112] mca: base: components_register: component tool register
> function successful
> [terra:387112] mca: base: components_register: found loaded component hnp
> [terra:387112] mca: base: components_register: component hnp has no
> register or open function
> [terra:387112] mca: base: components_register: found loaded component
> singleton
> [terra:387112] mca: base: components_register: component singleton
> register function successful
> [terra:387112] mca: base: components_open: opening ess components
> [terra:387112] mca: base: components_open: found loaded component slurm
> [terra:387112] mca: base: components_open: component slurm open function
> successful
> [terra:387112] mca: base: components_open: found loaded component env
> [terra:387112] mca: base: components_open: component env open function
> successful
> [terra:387112] mca: base: components_open: found loaded component pmi
> [terra:387112] mca: base: components_open: component pmi open function
> successful
> [terra:387112] mca: base: components_open: found loaded component tool
> [terra:387112] mca: base: components_open: component tool open function
> successful
> [terra:387112] mca: base: components_open: found loaded component hnp
> [terra:387112] mca: base: components_open: component hnp open function
> successful
> [terra:387112] mca: base: components_open: found loaded component singleton
> [terra:387112] mca: base: components_open: component singleton open
> function successful
> [terra:387112] mca:base:select: Auto-selecting ess components
> [terra:387112] mca:base:select:(  ess) Querying component [slurm]
> [terra:387112] mca:base:select:(  ess) Querying component [env]
> [terra:387112] mca:base:select:(  ess) Querying component [pmi]
> [terra:387112] mca:base:select:(  ess) Querying component [tool]
> [terra:387112] mca:base:select:(  ess) Querying component [hnp]
> [terra:387112] mca:base:select:(  ess) Query of component [hnp] set
> priority to 100
> [terra:387112] mca:base:select:(  ess) Querying component [singleton]
> [terra:387112] mca:base:select:(  ess) Selected component [hnp]
> [terra:387112] mca: base: close: component slurm closed
> [terra:387112] mca: base: close: unloading component slurm
> [terra:387112] mca: base: close: component env closed
> [terra:387112] mca: base: close: unloading component env
> [terra:387112] mca: base: close: component pmi closed
> [terra:387112] mca: base: close: unloading component pmi
> [terra:387112] mca: base: close: component tool closed
> [terra:387112] mca: base: close: unloading component tool
> [terra:387112] mca: base: close: component singleton closed
> [terra:387112] mca: base: close: unloading component singleton
> --
> It looks like orte_init failed for some reason; your parallel process is
> likely to abort.  There are many reasons that a parallel process can
> fail during orte_init; some of which are due to configuration or
> environment problems.  This failure appears to be an internal failure;
> here's some additional information (which may only be 

Re: [OMPI devel] mpirun 4.1.0 segmentation fault

[Gilles Gouaillardet via devel]

Andrej,

that's odd, there should be a mca_pmix_pmix3x.so (assuming you built
with the internal pmix)

what was your exact configure command line?

fwiw, in your build tree, there should be a
opal/mca/pmix/pmix3x/.libs/mca_pmix_pmix3x.so
if it's there, try running
sudo make install
once more and see if it helps

Cheers,

Gilles

On Mon, Feb 1, 2021 at 11:05 PM Andrej Prsa via devel
 wrote:
>
> Hi Gilles,
>
> > it seems only flux is a PMIx option, which is very suspicious.
> >
> > can you check other components are available?
> >
> > ls -l /usr/local/lib/openmpi/mca_pmix_*.so
>
> andrej@terra:~/system/tests/MPI$ ls -l /usr/local/lib/openmpi/mca_pmix_*.so
> -rwxr-xr-x 1 root root 97488 Feb  1 08:20
> /usr/local/lib/openmpi/mca_pmix_flux.so
> -rwxr-xr-x 1 root root 92240 Feb  1 08:20
> /usr/local/lib/openmpi/mca_pmix_isolated.so
>
> Thank you for your continued help!
>
> Cheers,
> Andrej
>

Re: [OMPI devel] mpirun 4.1.0 segmentation fault

[Gilles Gouaillardet via devel]

Andrej,

it seems only flux is a PMIx option, which is very suspicious.

can you check other components are available?

ls -l /usr/local/lib/openmpi/mca_pmix_*.so

will list them.

Cheers,

Gilles

On Mon, Feb 1, 2021 at 10:53 PM Andrej Prsa via devel
 wrote:
>
> Hi Gilles,
>
> > what is your mpirun command line?
> > is mpirun invoked from a batch allocation?
>
> I call mpirun directly; here's a full output:
>
> andrej@terra:~/system/tests/MPI$ mpirun --mca ess_base_verbose 10 --mca
> pmix_base_verbose 10 -np 4 python testmpi.py
> [terra:203257] mca: base: components_register: registering framework ess
> components
> [terra:203257] mca: base: components_register: found loaded component slurm
> [terra:203257] mca: base: components_register: component slurm has no
> register or open function
> [terra:203257] mca: base: components_register: found loaded component env
> [terra:203257] mca: base: components_register: component env has no
> register or open function
> [terra:203257] mca: base: components_register: found loaded component pmi
> [terra:203257] mca: base: components_register: component pmi has no
> register or open function
> [terra:203257] mca: base: components_register: found loaded component tool
> [terra:203257] mca: base: components_register: component tool register
> function successful
> [terra:203257] mca: base: components_register: found loaded component hnp
> [terra:203257] mca: base: components_register: component hnp has no
> register or open function
> [terra:203257] mca: base: components_register: found loaded component
> singleton
> [terra:203257] mca: base: components_register: component singleton
> register function successful
> [terra:203257] mca: base: components_open: opening ess components
> [terra:203257] mca: base: components_open: found loaded component slurm
> [terra:203257] mca: base: components_open: component slurm open function
> successful
> [terra:203257] mca: base: components_open: found loaded component env
> [terra:203257] mca: base: components_open: component env open function
> successful
> [terra:203257] mca: base: components_open: found loaded component pmi
> [terra:203257] mca: base: components_open: component pmi open function
> successful
> [terra:203257] mca: base: components_open: found loaded component tool
> [terra:203257] mca: base: components_open: component tool open function
> successful
> [terra:203257] mca: base: components_open: found loaded component hnp
> [terra:203257] mca: base: components_open: component hnp open function
> successful
> [terra:203257] mca: base: components_open: found loaded component singleton
> [terra:203257] mca: base: components_open: component singleton open
> function successful
> [terra:203257] mca:base:select: Auto-selecting ess components
> [terra:203257] mca:base:select:(  ess) Querying component [slurm]
> [terra:203257] mca:base:select:(  ess) Querying component [env]
> [terra:203257] mca:base:select:(  ess) Querying component [pmi]
> [terra:203257] mca:base:select:(  ess) Querying component [tool]
> [terra:203257] mca:base:select:(  ess) Querying component [hnp]
> [terra:203257] mca:base:select:(  ess) Query of component [hnp] set
> priority to 100
> [terra:203257] mca:base:select:(  ess) Querying component [singleton]
> [terra:203257] mca:base:select:(  ess) Selected component [hnp]
> [terra:203257] mca: base: close: component slurm closed
> [terra:203257] mca: base: close: unloading component slurm
> [terra:203257] mca: base: close: component env closed
> [terra:203257] mca: base: close: unloading component env
> [terra:203257] mca: base: close: component pmi closed
> [terra:203257] mca: base: close: unloading component pmi
> [terra:203257] mca: base: close: component tool closed
> [terra:203257] mca: base: close: unloading component tool
> [terra:203257] mca: base: close: component singleton closed
> [terra:203257] mca: base: close: unloading component singleton
> [terra:203257] mca: base: components_register: registering framework
> pmix components
> [terra:203257] mca: base: components_register: found loaded component flux
> [terra:203257] mca: base: components_register: component flux register
> function successful
> [terra:203257] mca: base: components_open: opening pmix components
> [terra:203257] mca: base: components_open: found loaded component flux
> [terra:203257] mca:base:select: Auto-selecting pmix components
> [terra:203257] mca:base:select:( pmix) Querying component [flux]
> [terra:203257] mca:base:select:( pmix) No component selected!
> [terra:203257] [[47344,0],0] ORTE_ERROR_LOG: Not found in file
> ess_hnp_module.c at line 320
> --
> It looks like orte_init failed for some reason; your parallel process is
> likely to abort.  There are many reasons that a parallel process can
> fail during orte_init; some of which are due to configuration or
> environment problems.  This failure appears to be an internal failure;
> here's 

Re: [OMPI devel] mpirun 4.1.0 segmentation fault

[Gilles Gouaillardet via devel]

Andrej,

what is your mpirun command line?
is mpirun invoked from a batch allocation?

in order to get some more debug info, you can

mpirun --mca ess_base_verbose 10 --mca pmix_base_verbose 10 ...


Cheers,

Gilles

On Mon, Feb 1, 2021 at 10:27 PM Andrej Prsa via devel
 wrote:
>
> Hi Gilles,
>
> > I invite you to do some cleanup
> > sudo rm -rf /usr/local/lib/openmpi /usr/local/lib/pmix
> > and then
> > sudo make install
> > and try again.
>
> Good catch! Alright, I deleted /usr/local/lib/openmpi and
> /usr/local/lib/pmix, then I rebuilt (make clean; make) and installed
> pmix from the latest master (should I use 3.1.6 instead?), and rebuilt
> (make clean; make) and installed the debug-enabled version of openmpi.
> Now I'm getting this:
>
> [terra:199344] [[43961,0],0] ORTE_ERROR_LOG: Not found in file
> ess_hnp_module.c at line 320
> --
> It looks like orte_init failed for some reason; your parallel process is
> likely to abort.  There are many reasons that a parallel process can
> fail during orte_init; some of which are due to configuration or
> environment problems.  This failure appears to be an internal failure;
> here's some additional information (which may only be relevant to an
> Open MPI developer):
>
>opal_pmix_base_select failed
>--> Returned value Not found (-13) instead of ORTE_SUCCESS
> --
>
> > if the issue persists, please post the output of the following commands
> > $ env | grep ^OPAL_
> > $ env | grep ^PMIX_
>
> I don't have any env variables defined.
>
> Cheers,
> Andrej
>

Re: [OMPI devel] mpirun 4.1.0 segmentation fault

[Gilles Gouaillardet via devel]

Andrej,

The log you just posted strongly suggests a previously built (without
--enable-debug) internal PMIx is being used.

I invite you to do some cleanup
sudo rm -rf /usr/local/lib/openmpi /usr/local/lib/pmix
and then
sudo make install
and try again.

if the issue persists, please post the output of the following commands
$ env | grep ^OPAL_
$ env | grep ^PMIX_

Cheers,

Gilles


On Mon, Feb 1, 2021 at 2:11 PM Andrej Prsa via devel
 wrote:
>
> Hi Ralph,
>
> > Just trying to understand - why are you saying this is a pmix problem? 
> > Obviously, something to do with mpirun is failing, but I don't see any 
> > indication here that it has to do with pmix.
>
> No -- 4.0.3 had the pmix problem -- whenever I tried to submit jobs
> across multiple nodes using slurm (i.e. -mca plm slurm), I'd get this:
>
> --
> An ORTE daemon has unexpectedly failed after launch and before
> communicating back to mpirun. This could be caused by a number
> of factors, including an inability to create a connection back
> to mpirun due to a lack of common network interfaces and/or no
> route found between them. Please check network connectivity
> (including firewalls and network routing requirements).
> --
>
> But the same would work if I submitted it with rsh (i.e. -mca plm rsh).
> I read online that there were issues with cpu bind so I thought 4.1.0
> might have resolved it.
>
> So, back to the problem at hand. I reconfigured with --enable-debug and
> this is what I get:
>
> andrej@terra:~/system/openmpi-4.1.0$ mpirun
> [terra:4145441] *** Process received signal ***
> [terra:4145441] Signal: Segmentation fault (11)
> [terra:4145441] Signal code:  (128)
> [terra:4145441] Failing at address: (nil)
> [terra:4145441] [ 0]
> /lib/x86_64-linux-gnu/libc.so.6(+0x46210)[0x7f487ebf4210]
> [terra:4145441] [ 1]
> /usr/local/lib/openmpi/mca_pmix_pmix3x.so(opal_pmix_pmix3x_check_evars+0x15c)[0x7f487a340b3c]
> [terra:4145441] [ 2]
> /usr/local/lib/openmpi/mca_pmix_pmix3x.so(pmix3x_server_init+0x496)[0x7f487a3422e6]
> [terra:4145441] [ 3]
> /usr/local/lib/libopen-rte.so.40(pmix_server_init+0x5da)[0x7f487ef2f5ec]
> [terra:4145441] [ 4]
> /usr/local/lib/openmpi/mca_ess_hnp.so(+0x58d5)[0x7f487e90a8d5]
> [terra:4145441] [ 5]
> /usr/local/lib/libopen-rte.so.40(orte_init+0x354)[0x7f487efab836]
> [terra:4145441] [ 6]
> /usr/local/lib/libopen-rte.so.40(orte_submit_init+0x123b)[0x7f487efad0cd]
> [terra:4145441] [ 7] mpirun(+0x16bc)[0x55d26c3bb6bc]
> [terra:4145441] [ 8] mpirun(+0x134d)[0x55d26c3bb34d]
> [terra:4145441] [ 9]
> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0x7f487ebd50b3]
> [terra:4145441] [10] mpirun(+0x126e)[0x55d26c3bb26e]
> [terra:4145441] *** End of error message ***
> Segmentation fault (core dumped)
>
> gdb backtrace:
>
> (gdb) r
> Starting program: /usr/local/bin/mpirun
> [Thread debugging using libthread_db enabled]
> Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
>
> Program received signal SIGSEGV, Segmentation fault.
> 0x73302b3c in opal_pmix_pmix3x_check_evars () from
> /usr/local/lib/openmpi/mca_pmix_pmix3x.so
> (gdb) bt
> #0  0x73302b3c in opal_pmix_pmix3x_check_evars () from
> /usr/local/lib/openmpi/mca_pmix_pmix3x.so
> #1  0x733042e6 in pmix3x_server_init () from
> /usr/local/lib/openmpi/mca_pmix_pmix3x.so
> #2  0x77ef15ec in pmix_server_init () at
> orted/pmix/pmix_server.c:296
> #3  0x778cc8d5 in rte_init () at ess_hnp_module.c:329
> #4  0x77f6d836 in orte_init (pargc=0x7fffddbc,
> pargv=0x7fffddb0, flags=4) at runtime/orte_init.c:271
> #5  0x77f6f0cd in orte_submit_init (argc=1, argv=0x7fffe478,
> opts=0x0) at orted/orted_submit.c:570
> #6  0x56bc in orterun (argc=1, argv=0x7fffe478) at
> orterun.c:136
> #7  0x534d in main (argc=1, argv=0x7fffe478) at main.c:13
>
> This build is using the latest openpmix from github master.
>
> Thanks,
> Andrej
>

Re: [OMPI devel] How to build Open MPI so the UCX used can be changed at runtime?

[Gilles Gouaillardet via devel]

Jeff,


What I meant is the default Linux behavior is to first lookup 
dependencies in the rpath, and then fallback to LD_LIBRARY_PATH


*unless* -Wl,--enable-new-dtags was used at link time.


In the case of Open MPI, -Wl,--enable-new-dtags is added to the MPI 
wrappers,


but Open MPI is *not* built with this option.


That means, that by default, mca_pml_ucx.so and friends will get 
libuc?.so libraries at runtime from rpath


(and that cannot be overridden by LD_LIBRARY_PATH).


Cheers,


Gilles


On 1/28/2021 12:52 AM, Jeff Squyres (jsquyres) wrote:
On Jan 27, 2021, at 2:00 AM, Gilles Gouaillardet via devel 
mailto:devel@lists.open-mpi.org>> wrote:


Tim,

a simple option is to

configure ... LDFLAGS="-Wl,--enable-new-dtags"


If Open MPI is built with this option, then LD_LIBRARY_PATH takes 
precedence over rpath


(the default is the opposite as correctly pointed by Yossi in an 
earlier message)


Are you sure about the default?  I just did a default Open MPI v4.1.0 
build on Linux with gcc 8.x:


$ mpicc --showme
gcc -I/home/jsquyres/bogus/include -pthread -Wl,-rpath 
-Wl,/home/jsquyres/bogus/lib -Wl,--enable-new-dtags 
-L/home/jsquyres/bogus/lib -lmpi


--
Jeff Squyres
jsquy...@cisco.com <mailto:jsquy...@cisco.com>

Re: [OMPI devel] How to build Open MPI so the UCX used can be changed at runtime?

[Gilles Gouaillardet via devel]

 [ucp/api/ucp.h],
 [ucp],

Oddly, the Open MPI configure script already has overrides for
ucx_CFLAGS, ucx_LIBS, and ucx_STATIC_LIBS, but nothing for something
like "ucx_LDFLAGS=''".
I didn't see a simple way to add support for such an override without
some more extensive changes to multiple m4 files.

On Sun, Jan 24, 2021 at 7:08 PM Gilles Gouaillardet via devel
 wrote:

Tim,

Have you tried using LD_LIBRARY_PATH?
I guess "hardcoding the full path" means "link with -rpath", and IIRC,
LD_LIBRARY_PATH
overrides this setting.


If this does not work, here something you can try (disclaimer: I did not)

export LD_LIBRARY_PATH=/same/install/prefix/ucx/1.9.0/lib
configure ... --with-ucx
CPPFLAGS=-I/same/install/prefix/ucx/1.9.0/include
LDFLAGS=-L/same/install/prefix/ucx/1.9.0/lib

I expect the UCX components use libuct.so instead of
/same/install/prefix/ucx/1.9.0/lib/libuct.so.
If your users want the debug version, then you can simply change your
LD_LIBRARY_PATH
(module swap ucx should do the trick)

Three caveats you should keep in mind:
  - it is your responsibility to ensure the debug and prod versions of
UCX are ABI compatible
  - it will be mandatory to load a ucx module (otherwise Open MPI won't
find UCX libraries)
  - this is a guess and I did not test this.


An other option (I did not try) would be to install UCX on your build
machine in /usr
(since I expect /usr/lib/libuct.so is not hardcoded) and then use
LD_LIBRARY_PATH
(I assume your ucx module set it) to point to the UCX flavor of your choice).

Cheers,

Gilles

On Mon, Jan 25, 2021 at 7:43 AM Tim Mattox via devel
 wrote:

I'm specifically wanting my users to be able to load a "debug" vs.
"tuned" UCX module, without me having to make two different Open MPI
installs... the combinatorics get bad after a few versions (I'm
already having multiple versions of Open MPI to handle the differences
in Fortran mpi mod files for various compilers.)
Here are the differences in the configure options between the two UCX modules:
debug version: --enable-logging --enable-debug --enable-assertions
--enable-params-check --prefix=/same/install/prefix/ucx/1.9.0/debug
tuned version: --disable-logging --disable-debug --disable-assertions
--disable-params-check --prefix=/same/install/prefix/ucx/1.9.0/tuned

We noticed that the --enable-debug option for UCX has a pretty
dramatic performance hit for one application (so far).
I've already tested that everything works fine if I replace UCX's .so
files manually in the filesystem, and the "new/changed" ones get
loaded, but a user can't make that kind of swap.
My hope is a user could type "module swap ucx/1.9.0/tuned
ucx/1.9.0/debug" when they want to enable debugging at the UCX layer.

On Sun, Jan 24, 2021 at 4:43 PM Yossi Itigin  wrote:

Hi,

One option is to use LD_PRELOAD to load all ucx libraries from a specific 
location
For example: mpirun -x 
LD_PRELOAD=::: 
...  

BTW, what is different about the other UCX configuration? Maybe this is 
something which can be resolved another way.

--Yossi

-Original Message-
From: devel  On Behalf Of Tim Mattox via devel
Sent: Sunday, 24 January 2021 23:18
To: devel@lists.open-mpi.org
Cc: Tim Mattox 
Subject: [OMPI devel] How to build Open MPI so the UCX used can be changed at 
runtime?

Hello,
I've run into an application that has its performance dramatically affected by 
some configuration options to the underlying UCX library.
Is there a way to configure/build Open MPI so that which UCX library is used is 
determined at runtime (e.g. by an environment module), rather than having to 
configure/build different instances of Open MPI?

When I configure Open MPI 4.1.0 with "--with-ucx" it is hardcoding the full 
path to the UCX .so library files to the UCX version it found at configure time.
--
Tim Mattox, Ph.D. - tmat...@gmail.com



--
Tim Mattox, Ph.D. - tmat...@gmail.com



--
Tim Mattox, Ph.D. - tmat...@gmail.com



--
Tim Mattox, Ph.D. - tmat...@gmail.com

Re: [OMPI devel] How to build Open MPI so the UCX used can be changed at runtime?

[Gilles Gouaillardet via devel]

Tim,

Have you tried using LD_LIBRARY_PATH?
I guess "hardcoding the full path" means "link with -rpath", and IIRC,
LD_LIBRARY_PATH
overrides this setting.


If this does not work, here something you can try (disclaimer: I did not)

export LD_LIBRARY_PATH=/same/install/prefix/ucx/1.9.0/lib
configure ... --with-ucx
CPPFLAGS=-I/same/install/prefix/ucx/1.9.0/include
LDFLAGS=-L/same/install/prefix/ucx/1.9.0/lib

I expect the UCX components use libuct.so instead of
/same/install/prefix/ucx/1.9.0/lib/libuct.so.
If your users want the debug version, then you can simply change your
LD_LIBRARY_PATH
(module swap ucx should do the trick)

Three caveats you should keep in mind:
 - it is your responsibility to ensure the debug and prod versions of
UCX are ABI compatible
 - it will be mandatory to load a ucx module (otherwise Open MPI won't
find UCX libraries)
 - this is a guess and I did not test this.


An other option (I did not try) would be to install UCX on your build
machine in /usr
(since I expect /usr/lib/libuct.so is not hardcoded) and then use
LD_LIBRARY_PATH
(I assume your ucx module set it) to point to the UCX flavor of your choice).

Cheers,

Gilles

On Mon, Jan 25, 2021 at 7:43 AM Tim Mattox via devel
 wrote:
>
> I'm specifically wanting my users to be able to load a "debug" vs.
> "tuned" UCX module, without me having to make two different Open MPI
> installs... the combinatorics get bad after a few versions (I'm
> already having multiple versions of Open MPI to handle the differences
> in Fortran mpi mod files for various compilers.)
> Here are the differences in the configure options between the two UCX modules:
> debug version: --enable-logging --enable-debug --enable-assertions
> --enable-params-check --prefix=/same/install/prefix/ucx/1.9.0/debug
> tuned version: --disable-logging --disable-debug --disable-assertions
> --disable-params-check --prefix=/same/install/prefix/ucx/1.9.0/tuned
>
> We noticed that the --enable-debug option for UCX has a pretty
> dramatic performance hit for one application (so far).
> I've already tested that everything works fine if I replace UCX's .so
> files manually in the filesystem, and the "new/changed" ones get
> loaded, but a user can't make that kind of swap.
> My hope is a user could type "module swap ucx/1.9.0/tuned
> ucx/1.9.0/debug" when they want to enable debugging at the UCX layer.
>
> On Sun, Jan 24, 2021 at 4:43 PM Yossi Itigin  wrote:
> >
> > Hi,
> >
> > One option is to use LD_PRELOAD to load all ucx libraries from a specific 
> > location
> > For example: mpirun -x 
> > LD_PRELOAD=:::
> >  ...  
> >
> > BTW, what is different about the other UCX configuration? Maybe this is 
> > something which can be resolved another way.
> >
> > --Yossi
> >
> > -Original Message-
> > From: devel  On Behalf Of Tim Mattox via 
> > devel
> > Sent: Sunday, 24 January 2021 23:18
> > To: devel@lists.open-mpi.org
> > Cc: Tim Mattox 
> > Subject: [OMPI devel] How to build Open MPI so the UCX used can be changed 
> > at runtime?
> >
> > Hello,
> > I've run into an application that has its performance dramatically affected 
> > by some configuration options to the underlying UCX library.
> > Is there a way to configure/build Open MPI so that which UCX library is 
> > used is determined at runtime (e.g. by an environment module), rather than 
> > having to configure/build different instances of Open MPI?
> >
> > When I configure Open MPI 4.1.0 with "--with-ucx" it is hardcoding the full 
> > path to the UCX .so library files to the UCX version it found at configure 
> > time.
> > --
> > Tim Mattox, Ph.D. - tmat...@gmail.com
>
>
>
> --
> Tim Mattox, Ph.D. - tmat...@gmail.com

Re: [OMPI devel] Which compiler versions to test?

[Gilles Gouaillardet via devel]

Hi Jeff,

On RHEL 8.x, the default gcc compiler is 8.3.1, so I think it is worth testing.

Containers could be used to setup a RHEL 8.x environment (so not only
gcc but also third party libs such as libevent and hwloc can be used)
if the MTT cluster will not shrink bigger.

Cheers,

Gilles

On Fri, Oct 9, 2020 at 1:28 AM Jeff Squyres (jsquyres) via devel
 wrote:
>
> Open question to the Open MPI dev community...
>
> Over time, the size of my MTT cluster has been growing smaller (due to 
> hardware failure, power budget restrictions, etc.).  This means that I have 
> far fewer CPU cycles available for testing various compilers and configure 
> CLI options than I used to.
>
> What compilers does the community think are worthwhile to test these days?  I 
> generally have access to gcc/gfortran, clang, and some versions of the Intel 
> compiler suite.
>
> master, 4.0.x, and 4.1.x branches
> - gcc 4.8.5 (i.e., the default gcc on RHEL 7.x)
> - gcc 9.latest
> - gcc 10.latest
> - clang 9.0.latest
> - clang 10.0.latest
> - Intel 2017
> - Intel 2019
>
> (I don't have Intel 2018 or Intel 2020)
>
> Is this sufficient?  Or is it worthwhile to test other versions of these 
> compilers?
>
> --
> Jeff Squyres
> jsquy...@cisco.com
>

Re: [OMPI devel] Hanging inside ompi_win_create_dynamic

[Gilles Gouaillardet via devel]

Luis,

a MPI window creation is a collective operation, that involves the
creation of a communicator that involves (several) non blocking
allgather.

I recommend you first make sure you that creating a windows does not
start a recursion.


Cheers,

Gilles

On Fri, May 22, 2020 at 7:53 PM Luis Cebamanos via devel
 wrote:
>
> Hello OpenMPI devs,
>
> I know this can be a bit vague question, but I am looking for some hint
> that could help me to debug the problem I am facing here.
>
> I am trying to create a dynamic window inside a collective operation.
> Inside the operation, the call to ompi_win_create_dynamic hangs with
> processes waiting in ompi_request_wait_completion:
>
> rc = ompi_comm_nextcid_nb (newcomm, comm, bridgecomm, arg0, arg1,
> send_first, mode, );
> if (OMPI_SUCCESS != rc) {
> return rc;
> }
>
> ompi_request_wait_completion (req);
>
> There is nothing obvious there but I can see from my debugging that
> processes end up in (opal_progress.c):
>
> #if OPAL_HAVE_SCHED_YIELD
> if (opal_progress_yield_when_idle && events <= 0) {
> /* If there is nothing to do - yield the processor - otherwise
>  * we could consume the processor for the entire time slice. If
>  * the processor is oversubscribed - this will result in a best-case
>  * latency equivalent to the time-slice.
>  */
> sched_yield();
> }
> #endif  /* defined(HAVE_SCHED_YIELD) */
>
> I have been debugging it for a while now and I am starting to feel like
> I am driving in circles. Is there anything I should look at in this
> particular situation? What could possible be the caused?
>
> Regards,
> Luis
>
>
> The University of Edinburgh is a charitable body, registered in Scotland, 
> with registration number SC005336.

Re: [OMPI devel] Warning

[Gilles Gouaillardet via devel]

Luis,

you can do this:

struct ompi_info_t * info = _mpi_info_null.info;
ret = ompi_win_create_dynamic(_null->super, comm, win);

Cheers,

Gilles

On Fri, May 15, 2020 at 7:38 PM Luis via devel  wrote:
>
> Hi OMPI devs,
>
> I was wondewring if this warning is expected, if not, how should we
> internally call ompi_win_create_dynamic?
>
> res = ompi_win_create_dynamic(MPI_INFO_NULL, comm, );
>  ^
> In file included from pnbc_osc_internal.h:40,
>  from pnbc_osc_iallreduce.c:21:
> ../../../../ompi/win/win.h:143:42: note: expected ‘opal_info_t *’ {aka
> ‘struct opal_info_t *’} but argument is of type ‘struct ompi_info_t *’
>  int ompi_win_create_dynamic(opal_info_t *info, ompi_communicator_t
> *comm, ompi_win_t **newwin);
>
>
> Regards,
> Luis
> The University of Edinburgh is a charitable body, registered in Scotland, 
> with registration number SC005336.

Re: [OMPI devel] Open MPI v4.0.1: Process is hanging inside MPI_Init() when debugged with TotalView

[Gilles Gouaillardet via devel]

John,

OMPI_LAZY_WAIT_FOR_COMPLETION(active)


is a simple loop that periodically checks the (volatile) "active" 
condition, that is expected to be updated by an other thread.
So if you set your breakpoint too early, and **all** threads are stopped 
when this breakpoint is hit, you might experience

what looks like a race condition.
I guess a similar scenario can occur if the breakpoint is set in 
mpirun/orted too early, and prevents the pmix (or oob/tcp) thread

from sending the message to all MPI tasks)



Ralph,

does the v4.0.x branch still need the oob/tcp progress thread running 
inside the MPI app?
or are we missing some commits (since all interactions with mpirun/orted 
are handled by PMIx, at least in the master branch) ?


Cheers,

Gilles

On 11/12/2019 9:27 AM, Ralph Castain via devel wrote:

Hi John

Sorry to say, but there is no way to really answer your question as 
the OMPI community doesn't actively test MPIR support. I haven't seen 
any reports of hangs during MPI_Init from any release series, 
including 4.x. My guess is that it may have something to do with the 
debugger interactions as opposed to being a true race condition.


Ralph


On Nov 8, 2019, at 11:27 AM, John DelSignore via devel 
mailto:devel@lists.open-mpi.org>> wrote:


Hi,

An LLNL TotalView user on a Mac reported that their MPI job was 
hanging inside MPI_Init() when started under the control of 
TotalView. They were using Open MPI 4.0.1, and TotalView was using 
the MPIR Interface (sorry, we don't support the PMIx debugging hooks 
yet).


I was able to reproduce the hang on my own Linux system with my own 
build of Open MPI 4.0.1, which I built with debug symbols. As far as 
I can tell, there is some sort of race inside of Open MPI 4.0.1, 
because if I placed breakpoints at certain points in the Open MPI 
code, and thus change the timing slightly, that was enough to avoid 
the hang.


When the code hangs, it appeared as if one or more MPI processes are 
waiting inside ompi_mpi_init() at line ompi_mpi_init.c#904 for a 
fence to be released. In one of the runs, rank 0 was the only one the 
was hanging there (though I have seen runs where two ranks were hung 
there).


Here's a backtrace of the first thread in the rank 0 process in the 
case where one rank was hung:


d1.<> f 10.1 w
>  0 __nanosleep_nocancel PC=0x774e2efd, FP=0x7fffd1e0 
[/lib64/libc.so.6]

   1 usleep PC=0x77513b2f, FP=0x7fffd200 [/lib64/libc.so.6]
   2 ompi_mpi_init PC=0x77a64009, FP=0x7fffd350 
[/home/jdelsign/src/tools-external/openmpi-4.0.1/ompi/runtime/ompi_mpi_init.c#904]
   3 PMPI_Init PC=0x77ab0be4, FP=0x7fffd390 
[/home/jdelsign/src/tools-external/openmpi-4.0.1-lid/ompi/mpi/c/profile/pinit.c#67]
   4 main PC=0x00400c5e, FP=0x7fffd550 
[/home/jdelsign/cpi.c#27]
   5 __libc_start_main PC=0x77446b13, FP=0x7fffd610 
[/lib64/libc.so.6]
   6 _start   PC=0x00400b04, FP=0x7fffd618 
[/amd/home/jdelsign/cpi]


Here's the block of code where the thread is hung:

    /* if we executed the above fence in the background, then
 * we have to wait here for it to complete. However, there
 * is no reason to do two barriers! */
    if (background_fence) {
OMPI_LAZY_WAIT_FOR_COMPLETION(active);
    } else if (!ompi_async_mpi_init) {
    /* wait for everyone to reach this point - this is a hard
 * barrier requirement at this time, though we hope to relax
 * it at a later point */
    if (NULL != opal_pmix.fence_nb) {
    active = true;
OPAL_POST_OBJECT();
    if (OMPI_SUCCESS != (ret = opal_pmix.fence_nb(NULL, false,
fence_release, (void* {
    error = "opal_pmix.fence_nb() failed";
    goto error;
    }
OMPI_LAZY_WAIT_FOR_COMPLETION(active); *- STUCK HERE WAITING 
FOR THE FENCE TO BE RELEASED*

    } else {
    if (OMPI_SUCCESS != (ret = opal_pmix.fence(NULL, false))) {
    error = "opal_pmix.fence() failed";
    goto error;
    }
    }
    }

And here is an aggregated backtrace of all of the processes and 
threads in the job:


d1.<> f g w -g f+l
+/
 +__clone : 5:12[0-3.2-3, p1.2-5]
 |+start_thread
 | +listen_thread@oob_tcp_listener.c 
#705 : 1:1[p1.5]

 | |+__select_nocancel
 | +listen_thread@ptl_base_listener.c 
#214 : 1:1[p1.3]

 | |+__select_nocancel
 | +progress_engine@opal_progress_threads.c 
#105 : 5:5[0-3.2, p1.4]
 | |+opal_libevent2022_event_base_loop@event.c 
#1632

 | | +poll_dispatch@poll.c #167
 | |  +__poll_nocancel
 | +progress_engine@pmix_progress_threads.c 
#108 : 5:5[0-3.3, p1.2]
 |  +opal_libevent2022_event_base_loop@event.c 
#1632

 |   

Re: [OMPI devel] hcoll missing libsharp

[Gilles Gouaillardet via devel]

Folks,


-L/path/to/lib -lfoo will look for the foo library (libfoo.so or 
libfoo.a on Linux) in the /path/to/lib directory.


In the case of a dynamic library (e.g. libfoo.so), its dependencies will 
*not* be searched in /path/to/lib


(and this is where LD_LIBRARY_PATH helps)


that can be evidenced with a quite simple test (libfoo.so depends on 
libbar.so, both in the current directory)


/* I run on an up-to-date CentOS 7 box, other linkers such as the ubuntu 
one might have a different behavior */



$ LD_LIBRARY_PATH=. ldd libfoo.so
    linux-vdso.so.1 =>  (0x7ffde9a3e000)
    libbar.so => ./libbar.so (0x7f793f5c9000)
    libc.so.6 => /lib64/libc.so.6 (0x7f793f1fb000)
    /lib64/ld-linux-x86-64.so.2 (0x7f793f9cd000)


$ gcc -o test test.c -L. -lfoo
/usr/bin/ld: warning: libbar.so, needed by ./libfoo.so, not found (try 
using -rpath or -rpath-link)

./libfoo.so: undefined reference to `bar'
collect2: error: ld returned 1 exit status


fails without LD_LIBRARY_PATH, but


$ LD_LIBRARY_PATH=. gcc -o test test.c -L. -lfoo

works.


FWIW, no LD_LIBRARY_PATH is needed at link time if the binary is *also* 
linked with libbar.so


(and this is *not* what is reported by config.log)


$ gcc -o test2 test2.c -L. -lfoo -lbar



Chris,


with LD_LIBRARY_PATH, are you facing the exact same error

(e.g. libsharp_coll.so.2 so cannot be found) ?

or are you only having an issue with undefined references in 
/lib64/libibnetdisc.so.5 ?



Cheers,


Gilles

On 10/16/2019 1:42 AM, Peter Kjellström via devel wrote:

On Tue, 15 Oct 2019 15:40:24 +
"Jeff Squyres \(jsquyres\) via devel"  wrote:
...

1) do you expect this to just work even if the linker has no idea
where to find libsharp.so ?

No, of course not.  :-)  That's what the additional -L was for.

LIBRARY_PATH is the corresponding variable for -L if one wants one.

This should work for both static and dynamic libraries as -L.
LD_LIBRARY_PATH on the other hand is run time for dynamic libraries
(configures the logic used by ld.so).

In a similar manner CPATH is an option to -I.

/Peter K

Re: [OMPI devel] hcoll missing libsharp

[Gilles GOUAILLARDET via devel]

Jeff,

I agree we are not trying to run the binary here, but I am still puzzled here...

Basically,

gcc -o a.out foo.o -L/.../hcoll/lib -lhcoll

and libhcoll.so depends on libsharp.so (that is in /.../sharp/lib)

1) do you expect this to just work even if the linker has no idea where to find 
libsharp.so ?
2) if not, how should the linker find libsharp.so without relying on 
LD_LIBRARY_PATH?

Cheers,

Gilles

On October 15, 2019, at 11:43 PM, "Jeff Squyres (jsquyres) via devel" 
 wrote:

You *shouldn't* need to set LD_LIBRARY_PATH, because configure is not trying to 
actually run the test app that it is compiling.  Indeed, it seems that the 
failure is occurring when it's trying to link (i.e., even if it was trying to 
run, it seems like the failure is before it would have tried to run). 


So the new compile line in config.log is:


configure:274185: gcc -std=gnu99 -std=gnu99 -o conftest -O3 -DNDEBUG

-finline-functions -fno-strict-aliasing -mcx16 -pthread

-I/install/u/tjcw/workspace/ompi/build/opal/mca/event/libevent2022/libevent/include

-I/install/u/tjcw/workspace/ompi/opal/mca/event/libevent2022/libevent

-I/install/u/tjcw/workspace/ompi/opal/mca/event/libevent2022/libevent/include

-I/install/u/tjcw/workspace/ompi/build/opal/mca/hwloc/hwloc201/hwloc/include

-I/install/u/tjcw/workspace/ompi/opal/mca/hwloc/hwloc201/hwloc/include

-I/install/u/tjcw/workspace/ibm_smpi_toucan_ucx/ompibase/dependencies/mofed_400/opt/mellanox/hcoll/include

-L/install/u/tjcw/workspace/ibm_smpi_toucan_ucx/ompibase/dependencies/mofed_400/opt/mellanox/sharp/lib

-L/install/u/tjcw/workspace/ibm_smpi_toucan_ucx/ompibase/dependencies/mofed_400/opt/mellanox/hcoll/lib

conftest.c -lhcoll -lrt -lm -lutil -lz >&5


Meaning: I see the additional -L that you put in there.


Just to confirm, the following command returns successfully, right?


ls -l 
/install/u/tjcw/workspace/ibm_smpi_toucan_ucx/ompibase/dependencies/mofed_400/opt/mellanox/sharp/lib/libsharp_coll.so.2


And if that returns successfully, what does this command show?


ldd 
/install/u/tjcw/workspace/ibm_smpi_toucan_ucx/ompibase/dependencies/mofed_400/opt/mellanox/sharp/lib/libsharp_coll.so.2


Finally, there are some other linker errors that are a little concerning, such 
as:


/lib64/libibnetdisc.so.5: undefined reference to `ibd_mkey'

/lib64/libibnetdisc.so.5: undefined reference to `get_mkey_by_portid'


I'm not sure what those are about / where those symbols will be located.





On Oct 15, 2019, at 10:07 AM, Chris Ward  wrote:


Adding an appropriate LDFLAGS= didn't help; the revised tarball is here 
http://tjcw.freeshell.org/ompi-output-2.tar.bz2 . Do I need to specify 
'-lsharp' to the link command ? If so, how do I do that ? 

T J (Chris) Ward, IBM Research. 
Scalable Data-Centric Computing - IBM Spectrum MPI 
IBM United Kingdom Ltd., Hursley Park, Winchester, Hants, SO21 2JN 
011-44-1962-818679 
LinkedIn https://www.linkedin.com/in/tjcward/ 
ResearchGate https://www.researchgate.net/profile/Thomas_Ward16 

IBM Research -- Data Centric Systems 
IBM Supercomputer Marketing 

IBM Branded Products IBM Branded Swag 




UNIX in the Cloud - Find A Place Where There's Room To Grow, with the original 
Open Standard. Free Trial Here Today Protein Folding by Supercomputer - 
BlueMatter Molecular Dynamics package. Data Tables - In-memory key-value store 
package. Linux on Windows - Virtualisation package. On the Lighthouse ! 

Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number 
741598. 
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU



-- 
Jeff Squyres
jsquy...@cisco.com 

Re: [OMPI devel] hcoll missing libsharp

[Gilles GOUAILLARDET via devel]

Chris,

I would try to add the directory containing libsharp in your LD_LIBRARY_PATH

Cheers,

Gilles

On October 15, 2019, at 11:08 PM, Chris Ward via devel 
 wrote:

Adding an appropriate LDFLAGS= didn't help; the revised tarball is here 
http://tjcw.freeshell.org/ompi-output-2.tar.bz2 . Do I need to specify 
'-lsharp' to the link command ? If so, how do I do that ? 

T J (Chris) Ward, IBM Research. 
Scalable Data-Centric Computing - IBM Spectrum MPI 
IBM United Kingdom Ltd., Hursley Park, Winchester, Hants, SO21 2JN 
011-44-1962-818679 
LinkedIn https://www.linkedin.com/in/tjcward/ 
ResearchGate https://www.researchgate.net/profile/Thomas_Ward16 

IBM Research -- Data Centric Systems 
IBM Supercomputer Marketing 

IBM Branded Products IBM Branded Swag 




UNIX in the Cloud - Find A Place Where There's Room To Grow, with the original 
Open Standard. Free Trial Here Today Protein Folding by Supercomputer - 
BlueMatter Molecular Dynamics package. Data Tables - In-memory key-value store 
package. Linux on Windows - Virtualisation package. On the Lighthouse ! 

Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number 
741598. 
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

Re: [OMPI devel] Is mpirun in slurm producing the correct "srun" cmd behind the scene?

[Gilles Gouaillardet via devel]

Lawrence,

My apologies for the incorrect link.
The patch is at https://github.com/open-mpi/ompi/pull/6672.patch

Cheers,

Gilles

On Sat, Jul 6, 2019 at 12:21 PM Gilles Gouaillardet
 wrote:
>
> Lawrence,
>
> this is a known issue (--cpu_bind optioned was removed from SLURM in
> favor of the --cpu-bind option) and the fix will be available in Open
> MPI 4.0.1
>
> Meanwhile, you can manually download and apply the patch at
> https://github.com/open-mpi/ompi/pull/6445.patch or use a nightly
> build of the v4.0.x branch.
>
> Cheers,
>
> Gilles
>
> On Sat, Jul 6, 2019 at 5:35 AM Sorrillo, Lawrence via devel
>  wrote:
> >
> > Hello,
> >
> >
> >
> > My attempt to run and troubleshoot my an OMPI job under a slurm allocation 
> > does not work as I would expect.
> >
> > The result below has led me to believe that under the hood, in this setup 
> > (SLURM with OMPI) that the correct srun options is not being used
> >
> > when I call, mpirun directly.
> >
> >
> >
> > Specifically the “cpu-bind=none” is breaking, but it also looks like the 
> > nodelist is incorrect.
> >
> >
> >
> > The job script.
> >
> >
> >
> >   1 #!/bin/bash
> >
> >   2
> >
> >   3 #SBATCH --job-name=mpi-hostname
> >
> >   4 #SBATCH --partition=dev
> >
> >   5 #SBATCH --account=Account1
> >
> >   6 #SBATCH --time=01:00:00
> >
> >   7 #SBATCH --nodes=2
> >
> >   8 #SBATCH --ntasks-per-node=1
> >
> >   9 #SBATCH --begin=now+10
> >
> > 10 #SBATCH --output="%x-%u-%N-%j.txt"  # 
> > jobName-userId-hostName-jobId.txt
> >
> > 11
> >
> > 12
> >
> > 13
> >
> > 14 # -- 
> > #
> >
> > 15 #module load DefApps
> >
> > 16 #module purge >/dev/null 2>&1
> >
> > 17 ##module load staging/slurm >/dev/null 2>&1
> >
> > 18 #module load  gcc/4.8.5 openmpi >/dev/null 2>&1
> >
> > 19 #module --ignore_cache spider openmpi/3.1.3 >/dev/null 2>&1
> >
> > 20 #
> >
> > 21 # -- 
> > #
> >
> > 22 #
> >
> > 23 MPI_RUN=$(which orterun)
> >
> > 24 if [[ -z "${MPI_RUN:+x}" ]]; then
> >
> > 25   echo "ERROR: Cannot find 'mpirun' executable..exiting"
> >
> > 26   exit 1
> >
> > 27 fi
> >
> > 28
> >
> > 29 echo
> >
> > 30 #CMD="orterun  -npernode 1 -np 2  /bin/hostname"
> >
> > 31 #CMD="srun /bin/hostname"
> >
> > 32 #CMD="srun -N2 -n2 --mpi=pmi2  /bin/hostname"
> >
> > 33 #MCMD="/sw/dev/openmpi401/bin/mpirun --bind-to-core  --report-bindings  
> > -mca btl openib,self -mca plm_base_verbose 10  /bin/hostname"
> >
> > 34 MCMD="/sw/dev/openmpi401/bin/mpirun  --report-bindings  -mca btl 
> > openib,self -mca plm_base_verbose 10  /bin/hostname"
> >
> > 35 echo "INFO: Executing the command: $MCMD"
> >
> > 36 $MCMD
> >
> > 37 sync
> >
> >
> >
> > Here is the output:
> >
> >
> >
> > 2 user1@node-login7g:~/git/slurm-jobs$ more 
> > mpi-hostname-user1-node513-835.txt
> >
> >   3
> >
> >   4 INFO: Executing the command: /sw/dev/openmpi401/bin/mpirun  
> > --report-bindings  -mca btl openib,self -mca plm_base_verbose 10  
> > /bin/hostname
> >
> >   5 [node513:32514] mca: base: components_register: registering framework 
> > plm components
> >
> >   6 [node513:32514] mca: base: components_register: found loaded component 
> > isolated
> >
> >   7 [node513:32514] mca: base: components_register: component isolated has 
> > no register or open function
> >
> >   8 [node513:32514] mca: base: components_register: found loaded component 
> > rsh
> >
> >   9 [node513:32514] mca: base: components_register: component rsh register 
> > function successful
> >
> > 10 [node513:32514] mca: base: components_register: found loaded component 
> > slurm
> >
> > 11 [node513:32514] mca: base: components_register: component slurm register 
> > function successful
> >
> > 12 [node513:32514] mca: base: components_register: found loaded component tm
> >
> > 13 [node513:32514] mca: base: components_register: component tm register 
> > function successful
> >
> > 14 [node513:32514] mca: base: components_open: opening plm components
> >
> > 15 [node513:32514] mca: base: components_open: found loaded component 
> > isolated
> >
> > 16 [node513:32514] mca: base: components_open: component isolated open 
> > function successful
> >
> > 17 [node513:32514] mca: base: components_open: found loaded component rsh
> >
> > 18 [node513:32514] mca: base: components_open: component rsh open function 
> > successful
> >
> > 19 [node513:32514] mca: base: components_open: found loaded component slurm
> >
> > 20 [node513:32514] mca: base: components_open: component slurm open 
> > function successful
> >
> > 21 [node513:32514] mca: base: components_open: found loaded component tm
> >
> > 22 [node513:32514] mca: base: components_open: component tm open function 
> > successful
> >
> > 23 [node513:32514] mca:base:select: Auto-selecting plm components
> >
> > 24 [node513:32514] mca:base:select:(  plm) Querying component [isolated]
> >
> > 25 [node513:32514] mca:base:select:(  plm) Query of component 

Re: [OMPI devel] Is mpirun in slurm producing the correct "srun" cmd behind the scene?

[Gilles Gouaillardet via devel]

Lawrence,

this is a known issue (--cpu_bind optioned was removed from SLURM in
favor of the --cpu-bind option) and the fix will be available in Open
MPI 4.0.1

Meanwhile, you can manually download and apply the patch at
https://github.com/open-mpi/ompi/pull/6445.patch or use a nightly
build of the v4.0.x branch.

Cheers,

Gilles

On Sat, Jul 6, 2019 at 5:35 AM Sorrillo, Lawrence via devel
 wrote:
>
> Hello,
>
>
>
> My attempt to run and troubleshoot my an OMPI job under a slurm allocation 
> does not work as I would expect.
>
> The result below has led me to believe that under the hood, in this setup 
> (SLURM with OMPI) that the correct srun options is not being used
>
> when I call, mpirun directly.
>
>
>
> Specifically the “cpu-bind=none” is breaking, but it also looks like the 
> nodelist is incorrect.
>
>
>
> The job script.
>
>
>
>   1 #!/bin/bash
>
>   2
>
>   3 #SBATCH --job-name=mpi-hostname
>
>   4 #SBATCH --partition=dev
>
>   5 #SBATCH --account=Account1
>
>   6 #SBATCH --time=01:00:00
>
>   7 #SBATCH --nodes=2
>
>   8 #SBATCH --ntasks-per-node=1
>
>   9 #SBATCH --begin=now+10
>
> 10 #SBATCH --output="%x-%u-%N-%j.txt"  # jobName-userId-hostName-jobId.txt
>
> 11
>
> 12
>
> 13
>
> 14 # -- #
>
> 15 #module load DefApps
>
> 16 #module purge >/dev/null 2>&1
>
> 17 ##module load staging/slurm >/dev/null 2>&1
>
> 18 #module load  gcc/4.8.5 openmpi >/dev/null 2>&1
>
> 19 #module --ignore_cache spider openmpi/3.1.3 >/dev/null 2>&1
>
> 20 #
>
> 21 # -- #
>
> 22 #
>
> 23 MPI_RUN=$(which orterun)
>
> 24 if [[ -z "${MPI_RUN:+x}" ]]; then
>
> 25   echo "ERROR: Cannot find 'mpirun' executable..exiting"
>
> 26   exit 1
>
> 27 fi
>
> 28
>
> 29 echo
>
> 30 #CMD="orterun  -npernode 1 -np 2  /bin/hostname"
>
> 31 #CMD="srun /bin/hostname"
>
> 32 #CMD="srun -N2 -n2 --mpi=pmi2  /bin/hostname"
>
> 33 #MCMD="/sw/dev/openmpi401/bin/mpirun --bind-to-core  --report-bindings  
> -mca btl openib,self -mca plm_base_verbose 10  /bin/hostname"
>
> 34 MCMD="/sw/dev/openmpi401/bin/mpirun  --report-bindings  -mca btl 
> openib,self -mca plm_base_verbose 10  /bin/hostname"
>
> 35 echo "INFO: Executing the command: $MCMD"
>
> 36 $MCMD
>
> 37 sync
>
>
>
> Here is the output:
>
>
>
> 2 user1@node-login7g:~/git/slurm-jobs$ more mpi-hostname-user1-node513-835.txt
>
>   3
>
>   4 INFO: Executing the command: /sw/dev/openmpi401/bin/mpirun  
> --report-bindings  -mca btl openib,self -mca plm_base_verbose 10  
> /bin/hostname
>
>   5 [node513:32514] mca: base: components_register: registering framework plm 
> components
>
>   6 [node513:32514] mca: base: components_register: found loaded component 
> isolated
>
>   7 [node513:32514] mca: base: components_register: component isolated has no 
> register or open function
>
>   8 [node513:32514] mca: base: components_register: found loaded component rsh
>
>   9 [node513:32514] mca: base: components_register: component rsh register 
> function successful
>
> 10 [node513:32514] mca: base: components_register: found loaded component 
> slurm
>
> 11 [node513:32514] mca: base: components_register: component slurm register 
> function successful
>
> 12 [node513:32514] mca: base: components_register: found loaded component tm
>
> 13 [node513:32514] mca: base: components_register: component tm register 
> function successful
>
> 14 [node513:32514] mca: base: components_open: opening plm components
>
> 15 [node513:32514] mca: base: components_open: found loaded component isolated
>
> 16 [node513:32514] mca: base: components_open: component isolated open 
> function successful
>
> 17 [node513:32514] mca: base: components_open: found loaded component rsh
>
> 18 [node513:32514] mca: base: components_open: component rsh open function 
> successful
>
> 19 [node513:32514] mca: base: components_open: found loaded component slurm
>
> 20 [node513:32514] mca: base: components_open: component slurm open function 
> successful
>
> 21 [node513:32514] mca: base: components_open: found loaded component tm
>
> 22 [node513:32514] mca: base: components_open: component tm open function 
> successful
>
> 23 [node513:32514] mca:base:select: Auto-selecting plm components
>
> 24 [node513:32514] mca:base:select:(  plm) Querying component [isolated]
>
> 25 [node513:32514] mca:base:select:(  plm) Query of component [isolated] set 
> priority to 0
>
> 26 [node513:32514] mca:base:select:(  plm) Querying component [rsh]
>
> 27 [node513:32514] mca:base:select:(  plm) Query of component [rsh] set 
> priority to 10
>
> 28 [node513:32514] mca:base:select:(  plm) Querying component [slurm]
>
> 29 [node513:32514] mca:base:select:(  plm) Query of component [slurm] set 
> priority to 75
>
> 30 [node513:32514] mca:base:select:(  plm) Querying component [tm]
>
> 31 [node513:32514] mca:base:select:(  plm) Selected component [slurm]
>
> 32 [node513:32514] mca: base: close: component isolated closed
>
>