Yes, thanks. I emailed the NERSc person and told him where we are and that we could fix this manually, but I don't have this issue on Summit ... He did not understand for the longest time, not sure he does now, that we do not add -gpu. And I gave him your reproducer. With any luck, with the reproducer, they can figure this out at some point. MPI is not working well for my scaling studies and my app has also given up on nvhpc, so we are good.
On Fri, Nov 5, 2021 at 9:49 AM Satish Balay <ba...@mcs.anl.gov> wrote: > For now - you could manually edit petscvariables and remove -mp=gpu from > it. > > Its primarily required to make configure happy. > > Satish > > > > On Thu, 4 Nov 2021, Barry Smith wrote: > > > > > This comes from the persistent problem with PETSc's make system using > too many flags for compiling CUDA that have not been tested by configure. > See below the -mp=gpu is provided probably from the CPPFLAGS or > CXXPPFLAGS(sp) that is improperly used by the PETSc makefiles to compile > CUDA code! > > > > > > Using CUDA compile: > /global/common/software/nersc/cos1.3/cuda/11.3.0/bin/nvcc -o .o > -I/opt/cray/pe/mpich/8.1.10/ofi/nvidia/20.7/include -I/opt/cray/pe/libsci/ > 21.08.1.2/NVIDIA/20.7/x86_64/include -I/opt/cray/pe/pmi/6.0.14/include > -I/opt/cray/pe/dsmml/0.2.2/dsmml//include > -I/opt/cray/xpmem/2.2.40-7.0.1.0_3.1__g1d7a24d.shasta/include -g > -Xcompiler -rdynamic -DLANDAU_DIM=2 -DLANDAU_MAX_SPECIES=10 > -DLANDAU_MAX_Q=4 -Xcompiler -fPIC -std=c++17 -gencode > arch=compute_80,code=sm_80 -Wno-deprecated-gpu-targets -c > --compiler-options=-I/global/homes/m/madams/petsc/include > -I/global/homes/m/madams/petsc/arch-perlmutter-opt-nvidia-cuda/include > -I/global/common/software/nersc/cos1.3/cuda/11.3.0/include -mp=gpu > -I/global/homes/m/madams/petsc/include > -I/global/homes/m/madams/petsc/arch-perlmutter-opt-nvidia-cuda/include > -I/global/common/software/nersc/cos1.3/cuda/11.3.0/include > > > > > > > > > On Nov 4, 2021, at 7:31 PM, Mark Adams <mfad...@lbl.gov> wrote: > > > > > > OK, configure is done. > > > Maybe I have too many -mp=gpu > > > > > > CUDAC > arch-perlmutter-opt-nvidia-cuda/obj/sys/classes/random/impls/curand/curand2.o > > > gcc: error: unrecognized command line option ‘-mp=gpu’; did you mean > ‘-mpku’? > > > > > > On Thu, Nov 4, 2021 at 5:51 PM Barry Smith <bsm...@petsc.dev <mailto: > bsm...@petsc.dev>> wrote: > > > > > > Need the same thing for the C++ preprocessor flag > > > > > > > > > > > >> On Nov 4, 2021, at 5:44 PM, Mark Adams <mfad...@lbl.gov <mailto: > mfad...@lbl.gov>> wrote: > > >> > > >> It gets a lot further. > > >> > > >> On Thu, Nov 4, 2021 at 5:32 PM Mark Adams <mfad...@lbl.gov <mailto: > mfad...@lbl.gov>> wrote: > > >> OK, sorry I missed the CPPFLAGS. It is running now. > > >> Thanks, > > >> > > >> On Thu, Nov 4, 2021 at 4:43 PM Satish Balay <ba...@mcs.anl.gov > <mailto:ba...@mcs.anl.gov>> wrote: > > >> Multiple e-mail threads on the same issue (: > > >> > > >> As suggested in my earlier thread - add -mp=gpu to both CPPFLAGS and > CFLAGS [or LDFLAGS] > > >> > > >> Satish > > >> > > >> ------- > > >> Executing: cc -o /tmp/petsc-Vvs8_T/config.setCompilers/conftest -g > -DLANDAU_DIM=2 -DLANDAU_MAX_SPECIES=10 -DLANDAU_MAX_Q=4 > /tmp/petsc-Vvs8_T/config.setCompilers/conftest.o > > >> Possible ERROR while running linker: > > >> stderr: > > >> nvc-Warning-The -gpu option has no effect unless a language-specific > option to enable GPU code generation is used (e.g.: -acc, -mp=gpu, -stdpar, > -cuda) > > >> > > >> > > >> On Thu, 4 Nov 2021, Mark Adams wrote: > > >> > > >> > It is CPPFLAGS. I seem to get the same behavior. > > >> > > > >> > FWIW, I did get this response from NERSc but I don't know how to > interpret > > >> > it. > > >> > > > >> > He seems to be saying that I don't need -mp=gpu for the device > compiler > > >> > (nvcc). He seems to think that I am adding -gpu. > > >> > > > >> > nvcc -- The device compiler does not need any of those flags > because it > > >> > already knows that it's being fed cuda code. The warning you're > seeing is > > >> > coming from nvc (which is the host / CPU side compiler) if you're > in the > > >> > PrgEnv-nvidia environment. You should not need to add -mp=gpu and > -cuda, > > >> > please just add the -cuda flag (to your host code) not to the > device code. > > >> > > > >> > I will try to talk with this guy again. > > >> > > > >> > Thanks, > > >> > > > >> > > > >> > On Thu, Nov 4, 2021 at 4:11 PM Barry Smith <bsm...@petsc.dev > <mailto:bsm...@petsc.dev>> wrote: > > >> > > > >> > > > > >> > > Yes, you need to use the CPPFLAGS which maybe called CPPCFLAGS > I am not > > >> > > sure > > >> > > > > >> > > > > >> > > On Nov 4, 2021, at 3:23 PM, Mark Adams <mfad...@lbl.gov <mailto: > mfad...@lbl.gov>> wrote: > > >> > > > > >> > > Ah, CCFLAGS does not seem to work. > > >> > > > > >> > > On Thu, Nov 4, 2021 at 3:07 PM Barry Smith <bsm...@petsc.dev > <mailto:bsm...@petsc.dev>> wrote: > > >> > > > > >> > >> > > >> > >> You have to pass in the flag to turn off the bitching about > -gpu to the > > >> > >> C preprocessor, not the C compiler. > > >> > >> > > >> > >> > > >> > >> stderr: > > >> > >> nvc-Warning-The -gpu option has no effect unless a > language-specific > > >> > >> option to enable GPU code generation is used (e.g.: -acc, > -mp=gpu, -stdpar, > > >> > >> -cuda) > > >> > >> Source: > > >> > >> #include "confdefs.h" > > >> > >> #include "conffix.h" > > >> > >> #include <stdlib.h> > > >> > >> > > >> > >> > > >> > >> > > >> > >> > On Nov 4, 2021, at 2:49 PM, Mark Adams <mfad...@lbl.gov > <mailto:mfad...@lbl.gov>> wrote: > > >> > >> > > > >> > >> > on Perlmutter with nvhpc: > > >> > >> > > > >> > >> > Defined make macro "CPP" to "cc --use cpp32" > > >> > >> > Preprocessing source: > > >> > >> > #include "confdefs.h" > > >> > >> > #include "conffix.h" > > >> > >> > #include <stdlib.h> > > >> > >> > > > >> > >> > Executing: cc --use cpp32 > -I/tmp/petsc-jV9U1b/config.setCompilers > > >> > >> /tmp/petsc-jV9U1b/config.setCompilers/conftest.c > > >> > >> > Possible ERROR while running preprocessor: exit code 1 > > >> > >> > stderr: > > >> > >> > nvc-Error-Unknown switch: --use > > >> > >> > Source: > > >> > >> > <configure.log> > > >> > >> > > >> > >> <configure.log> > > >> > > > > >> > > > > >> > > > > >> > > > >> > > >> <configure.log> > > > > > > <make.log><configure.log> > > > > >
