OK, starting clean.
OS CentOS 7.9 (7.9.2009)
mlnxofed 5.4-1.0.3.0
UCX 1.11.0 (from mlnxofed)
hcoll-4.7.3199 (from mlnxofed)
knem-1.1.4.90 (from mlnxofed)
nVidia HPC-SDK 21.9
OpenMPI 4.1.1
HWLOC 2.5.0
Straight configure
configure CC=nvc CXX=nvc++ FC=nvfortran
dies in
FCLD libmpi_usempif08.la
/usr/bin/ld: .libs/comm_spawn_multiple_f08.o: relocation R_X86_64_32S against `.rodata' can not be
used when making a shared object; recompile with -fPIC
configure CC=nvc CXX=nvc++ FC=nvfortran FCFLAGS='-fPIC'
Fixes that, dies in
CCLD mca_op_avx.la
./.libs/liblocal_ops_avx512.a(liblocal_ops_avx512_la-op_avx_functions.o):(.data+0x0): multiple
definition of `ompi_op_avx_functions_avx2'
./.libs/liblocal_ops_avx2.a(liblocal_ops_avx2_la-op_avx_functions.o):(.data+0x0):
first defined here
configure CC=nvc CXX=nvc++ FC=nvfortran FCFLAGS=-fPIC
--enable-mca-no-build=op-avx
succeeds
And, working up to what I really want, I can build (soemwhat emulating the
HPC-X 2.9.0 build)
/project/muno/OpenMPI/BUILD/SRC/openmpi-4.1.1/configure CC=nvc CXX=nvc++ FC=nvfortran FCFLAGS=-fPIC
--enable-mca-no-build=op-avx --prefix=/stage/opt/OpenMPI/ROME/4.1.1/NV-HPC/21.9
--with-libevent=internal --enable-mpi1-compatibility --without-xpmem --with-pmi --enable-mpi-cxx
--with-hwloc=/stage/opt/HWLOC/2.5.0 --with-hcoll=/opt/mellanox/hcoll
--with-knem=/opt/knem-1.1.4.90mlnx1 --with-cuda=/stage/opt/NV_hpc_sdk/Linux_x86_64/21.9/cuda
adding
--with-platform=/project/muno/OpenMPI/BUILD/SRC/openmpi-4.1.1/contrib/platform/mellanox/optimized
brings back
/usr/bin/ld: final link failed: Bad value
make[2]: *** [libmpi_usempif08.la] Error 2
Since it overrides the FCFLAGS command line setting apparently.
Editing the file to add -fPIC to FCLAGS took care of that as did the FC='nvfortran -fPIC' (which is
kludgey).
-Ray Muno
On 9/30/21 8:13 AM, Gilles Gouaillardet via users wrote:
Ray,
there is a typo, the configure option is
--enable-mca-no-build=op-avx
Cheers,
Gilles
----- Original Message -----
Added -*-enable-mca-no-build=op-avx *to the configure line. Still dies in
the same place.
history | grep config
CCLD mca_op_avx.la
./.libs/liblocal_ops_avx512.a(liblocal_ops_avx512_la-op_avx_functions.o):(.data+0x0):
multiple
definition of `ompi_op_avx_functions_avx2'
./.libs/liblocal_ops_avx2.a(liblocal_ops_avx2_la-op_avx_functions.o):(.data+0x0):
first defined here
./.libs/liblocal_ops_avx512.a(liblocal_ops_avx512_la-op_avx_functions.o):
In function
`ompi_op_avx_2buff_min_uint16_t_avx2':
/project/muno/OpenMPI/BUILD/SRC/openmpi-4.1.1/ompi/mca/op/avx/op_avx_functions.c:651:
multiple
definition of `ompi_op_avx_3buff_functions_avx2'
./.libs/liblocal_ops_avx2.a(liblocal_ops_avx2_la-op_avx_functions.o):/project/muno/OpenMPI/BUILD/SRC/openmpi-4.1.1/ompi/mca/op/avx/op_avx_functions.c:651:
first defined here
make[2]: *** [mca_op_avx.la] Error 2
make[2]: Leaving directory
`/project/muno/OpenMPI/BUILD/4.1.1/ROME/NV-HPC/21.9/ompi/mca/op/avx'
make[1]: *** [all-recursive] Error 1
make[1]: Leaving directory
`/project/muno/OpenMPI/BUILD/4.1.1/ROME/NV-HPC/21.9/ompi'
make: *** [all-recursive] Error 1
On 9/30/21 5:54 AM, Carl Ponder wrote:
For now, you can suppress this error building OpenMPI 4.1.1
./.libs/liblocal_ops_avx512.a(liblocal_ops_avx512_la-op_avx_functions.o):(.data+0x0):
multiple definition of `ompi_op_avx_functions_avx2'
./.libs/liblocal_ops_avx2.a(liblocal_ops_avx2_la-op_avx_functions.o):(.data+0x0):
first
defined here
./.libs/liblocal_ops_avx512.a(liblocal_ops_avx512_la-op_avx_functions.o): In
function
`ompi_op_avx_2buff_min_uint16_t_avx2':
/project/muno/OpenMPI/BUILD/SRC/openmpi-4.1.1/ompi/mca/op/avx/op_avx_functions.c:651:
multiple definition of `ompi_op_avx_3buff_functions_avx2'
./.libs/liblocal_ops_avx2.a(liblocal_ops_avx2_la-op_avx_functions.o):/project/muno/OpenMPI/BUILD/SRC/openmpi-4.1.1/ompi/mca/op/avx/op_avx_functions.c:651:
first defined here
with the NVHPC/PGI 21.9 compiler by using the setting
configure -*-enable-mca-no-build=op-avx* ...
We're still looking at the cause here. I don't have any advice about the
problem with 21.7.
----------------------------------------------------------------------------------------------------
Subject: Re: [OMPI users] OpenMPI 4.1.1, CentOS 7.9, nVidia HPC-SDk,
build hints?
Date: Wed, 29 Sep 2021 12:25:43 -0500
From: Ray Muno via users <users@lists.open-mpi.org>
Reply-To: Open MPI Users <users@lists.open-mpi.org>
To: users@lists.open-mpi.org
CC: Ray Muno <m...@aem.umn.edu>
External email: Use caution opening links or attachments
Tried this
configure CC='nvc -fPIC' CXX='nvc++ -fPIC' FC='nvfortran -fPIC'
Configure completes. Compiles quite a way through. Dies in a different
place. It does get past the
first error, however with libmpi_usempif08.la
FCLD libmpi_usempif08.la
make[2]: Leaving directory
`/project/muno/OpenMPI/BUILD/4.1.1/ROME/NV-HPC/21.9/ompi/mpi/fortran/use-mpi-f08'
Making all in mpi/fortran/mpiext-use-mpi-f08
make[2]: Entering directory
`/project/muno/OpenMPI/BUILD/4.1.1/ROME/NV-HPC/21.9/ompi/mpi/fortran/mpiext-use-mpi-f08'
PPFC mpi-f08-ext-module.lo
FCLD libforce_usempif08_module_to_be_built.la
make[2]: Leaving directory
`/project/muno/OpenMPI/BUILD/4.1.1/ROME/NV-HPC/21.9/ompi/mpi/fortran/mpiext-use-mpi-f08'
Dies here now.
CCLD liblocal_ops_avx512.la
CCLD mca_op_avx.la
./.libs/liblocal_ops_avx512.a(liblocal_ops_avx512_la-op_avx_functions.o):(.data+0x0):
multiple
definition of `ompi_op_avx_functions_avx2'
./.libs/liblocal_ops_avx2.a(liblocal_ops_avx2_la-op_avx_functions.o):(.data+0x0):
first
defined here
./.libs/liblocal_ops_avx512.a(liblocal_ops_avx512_la-op_avx_functions.o):
In function
`ompi_op_avx_2buff_min_uint16_t_avx2':
/project/muno/OpenMPI/BUILD/SRC/openmpi-4.1.1/ompi/mca/op/avx/op_avx_functions.c:651:
multiple
definition of `ompi_op_avx_3buff_functions_avx2'
./.libs/liblocal_ops_avx2.a(liblocal_ops_avx2_la-op_avx_functions.o):/project/muno/OpenMPI/BUILD/SRC/openmpi-4.1.1/ompi/mca/op/avx/op_avx_functions.c:651:
first defined here
make[2]: *** [mca_op_avx.la] Error 2
make[2]: Leaving directory
`/project/muno/OpenMPI/BUILD/4.1.1/ROME/NV-HPC/21.9/ompi/mca/op/avx'
make[1]: *** [all-recursive] Error 1
make[1]: Leaving directory
`/project/muno/OpenMPI/BUILD/4.1.1/ROME/NV-HPC/21.9/ompi'
make: *** [all-recursive] Error 1
On 9/29/21 11:42 AM, Bennet Fauber via users wrote:
Ray,
If all the errors about not being compiled with -fPIC are still appearing,
there may be a bug that
is preventing the option from getting through to the compiler(s). It might
be worth looking through
the logs to see the full compile command for one or more of them to see
whether that is true? Say,
libs/comm_spawn_multiple_f08.o for example?
If -fPIC is missing, you may be able to recompile that manually with the
-fPIC in place, then remake
and see if that also causes the link error to go away, that would be a good
start.
Hope this helps, -- bennet
On Wed, Sep 29, 2021 at 12:29 PM Ray Muno via users
<users@lists.open-mpi.org
<mailto:users@lists.open-mpi.org>> wrote:
I did try that and it fails at the same place.
Which version of the nVidia HPC-SDK are you using? I a m using 21.7. I
see there is an upgrade to
21.9, which came out since I installed. I have that installed and
will try to see if they changed
anything. Not much in the releases notes to indicate any major changes.
-Ray Muno
On 9/29/21 10:54 AM, Jing Gong wrote:
> Hi,
>
>
> Before Nvidia persons look into details,pProbably you can try to add the flag
"-fPIC" to the
> nvhpc compiler likes cc="nvc -fPIC", which at least worked with me.
>
>
>
> /Jing
>
>
----------------------------------------------------------------------------------------------------
> *From:* users <users-boun...@lists.open-mpi.org
<mailto:users-boun...@lists.open-mpi.org>> on
behalf of Ray Muno via users
> <users@lists.open-mpi.org
<mailto:users@lists.open-mpi.org>>
> *Sent:* Wednesday, September 29, 2021 17:22
> *To:* Open MPI User's List
> *Cc:* Ray Muno
> *Subject:* Re: [OMPI users] OpenMPI 4.1.1, CentOS 7.9, nVidia
HPC-SDk, build hints?
> Thanks, I looked through previous emails here in the user list.
Iguess I need to subscribe
to the
> Developers list.
>
> -Ray Muno
>
> On 9/29/21 9:58 AM, Jeff Squyres (jsquyres) wrote:
>> Ray --
>>
>> Looks like this is a dup
ofhttps://github.com/open-mpi/ompi/issues/8919
<https://github.com/open-mpi/ompi/issues/8919>
<https://github.com/open-mpi/ompi/issues/8919
<https://github.com/open-mpi/ompi/issues/8919>>
>> <https://github.com/open-mpi/ompi/issues/8919
<https://github.com/open-mpi/ompi/issues/8919>
<https://github.com/open-mpi/ompi/issues/8919
<https://github.com/open-mpi/ompi/issues/8919>>>.
>>
>>
>
--
Ray Muno
IT Systems Administrator
e-mail:m...@umn.edu
<mailto:m...@umn.edu>
University of Minnesota
Aerospace Engineering and Mechanics
--
Ray Muno
IT Systems Administrator
e-mail: m...@umn.edu
Phone: (612) 625-9531
University of Minnesota
Aerospace Engineering and Mechanics
110 Union St. S.E.
Minneapolis, MN 55455
--
Ray Muno
Computer Systems Administrator
e-mail:m...@aem.umn.edu
Phone: (612) 625-9531
University of Minnesota
Aerospace Engineering and Mechanics
110 Union St. S.E.
Minneapolis, MN 55455
--
Ray Muno
IT Systems Administrator
e-mail: m...@umn.edu
Phone: (612) 625-9531
University of Minnesota
Aerospace Engineering and Mechanics
110 Union St. S.E.
Minneapolis, MN 55455
