Yes, PETSc sends down a single -O3 to SuperLU_DIST CMake but internally SuperLU_DIST CMake is adding its own set of nvcc compiler flags to use and these that get added to the one's PETSc passes in (they do not replace them). A little further down in the file it displays a compile line with nvcc that has two sets of -O3 (as well as two sets of other things as well).
I think we changed the main branch PETSc to not pass down the optimization flags to the CMake CUDA to prevent the problem from happening. In other words, we let the package decide its optimization flags instead of trying to tell it what to use. Barry The root of the problem is that compiler APIs are a bunch of strings passed on the command line to the compiler without any structure. > On Aug 26, 2022, at 9:47 AM, byron <lbgpub...@gmail.com> wrote: > > Hi Barry > > I dont think I see the same thing. > > When I do a build using the release branch petsc passes the following > arguements to superlu_dist (taken from > /home/blahblah/Projects/220826-vanilla-petsc-release-branch/petsc/arch-linux-c-opt/externalpackages/git.superlu_dist/superlu_dist.petscconf) > which has only one -O3 entry > > -DCMAKE_INSTALL_PREFIX=/home/blahblah/Projects/220826-vanilla-petsc-release-branch/installation > > -DCMAKE_INSTALL_NAME_DIR:STRING="/home/lgoodman/Projects/220826-vanilla-petsc-release-branch/installation/lib" > -DCMAKE_INSTALL_LIBDIR:STRING="lib" -DCMAKE_VERBOSE_MAKEFILE=1 > -DCMAKE_BUILD_TYPE=Release -DCMAKE_C_COMPILER="mpicc" > -DMPI_C_COMPILER="/usr/bin/mpicc" -DCMAKE_AR=/usr/bin/ar > -DCMAKE_RANLIB=/usr/bin/ranlib -DCMAKE_C_FLAGS:STRING="-fPIC > -Wno-lto-type-mismatch -fstack-protector -g -O -fopenmp -DDEBUGlevel=0 > -DPRNTlevel=0" -DCMAKE_C_FLAGS_DEBUG:STRING="-fPIC -Wno-lto-type-mismatch > -fstack-protector -g -O -fopenmp -DDEBUGlevel=0 -DPRNTlevel=0" > -DCMAKE_C_FLAGS_RELEASE:STRING="-fPIC -Wno-lto-type-mismatch > -fstack-protector -g -O -fopenmp -DDEBUGlevel=0 -DPRNTlevel=0" > -DCMAKE_CXX_COMPILER="mpicxx" -DMPI_CXX_COMPILER="/usr/bin/mpicxx" > -DCMAKE_CXX_FLAGS:STRING="-Wno-lto-type-mismatch -fstack-protector -g -O > -fopenmp -std=c++11 -fPIC -fopenmp -DDEBUGlevel=0 -DPRNTlevel=0" > -DCMAKE_CXX_FLAGS_DEBUG:STRING="-Wno-lto-type-mismatch -fstack-protector -g > -O -fopenmp -std=c++11 -fPIC -fopenmp -DDEBUGlevel=0 -DPRNTlevel=0" > -DCMAKE_CXX_FLAGS_RELEASE:STRING="-Wno-lto-type-mismatch -fstack-protector -g > -O -fopenmp -std=c++11 -fPIC -fopenmp -DDEBUGlevel=0 -DPRNTlevel=0" > -DCMAKE_Fortran_COMPILER="mpif90" -DMPI_Fortran_COMPILER="/usr/bin/mpif90" > -DCMAKE_Fortran_FLAGS:STRING="-fPIC -ffree-line-length-0 > -Wno-lto-type-mismatch -g -O -fopenmp" > -DCMAKE_Fortran_FLAGS_DEBUG:STRING="-fPIC -ffree-line-length-0 > -Wno-lto-type-mismatch -g -O -fopenmp" > -DCMAKE_Fortran_FLAGS_RELEASE:STRING="-fPIC -ffree-line-length-0 > -Wno-lto-type-mismatch -g -O -fopenmp" -DCMAKE_EXE_LINKER_FLAGS:STRING=" > -fopenmp -fopenmp" -DBUILD_SHARED_LIBS:BOOL=ON -DTPL_ENABLE_CUDALIB=TRUE > -DTPL_CUDA_LIBRARIES="-lcudart -lcufft -lcublas -lcusparse -lcusolver > -lcurand -lcuda" -DCUDA_ARCH_FLAGS="-arch=sm_75" -DCMAKE_CUDA_COMPILER="nvcc" > -DCMAKE_CUDA_FLAGS=" -ccbin mpicxx -std=c++14 -Xcompiler -fPIC -O3 -gencode > arch=compute_75,code=sm_75 -Wno-deprecated-gpu-targets > -I/usr/lib/x86_64-linux-gnu/openmpi/include/openmpi > -I/usr/lib/x86_64-linux-gnu/openmpi/include -DDEBUGlevel=0 -DPRNTlevel=0" > -DUSE_XSDK_DEFAULTS=YES > -DTPL_BLAS_LIBRARIES="-Wl,-rpath,/home/lgoodman/Projects/220826-vanilla-petsc-release-branch/installation/lib > > -L/home/lgoodman/Projects/220826-vanilla-petsc-release-branch/installation/lib > -lflapack > -Wl,-rpath,/home/lgoodman/Projects/220826-vanilla-petsc-release-branch/installation/lib > > -L/home/lgoodman/Projects/220826-vanilla-petsc-release-branch/installation/lib > -lfblas -lcudart -lcufft -lcublas -lcusparse -lcusolver -lcurand -lcuda -lm > -lstdc++ -ldl -Wl,-rpath,/usr/lib/x86_64-linux-gnu/openmpi/lib > -L/usr/lib/x86_64-linux-gnu/openmpi/lib -lmpi_usempif08 > -lmpi_usempi_ignore_tkr -lmpi_mpifh -lmpi -lgfortran -lm > -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/9 -L/usr/lib/gcc/x86_64-linux-gnu/9 > -lgfortran -lm -lgcc_s -lquadmath -lpthread" > -DTPL_LAPACK_LIBRARIES="-Wl,-rpath,/home/lgoodman/Projects/220826-vanilla-petsc-release-branch/installation/lib > > -L/home/lgoodman/Projects/220826-vanilla-petsc-release-branch/installation/lib > -lflapack > -Wl,-rpath,/home/lgoodman/Projects/220826-vanilla-petsc-release-branch/installation/lib > > -L/home/lgoodman/Projects/220826-vanilla-petsc-release-branch/installation/lib > -lfblas -lcudart -lcufft -lcublas -lcusparse -lcusolver -lcurand -lcuda -lm > -lstdc++ -ldl -Wl,-rpath,/usr/lib/x86_64-linux-gnu/openmpi/lib > -L/usr/lib/x86_64-linux-gnu/openmpi/lib -lmpi_usempif08 > -lmpi_usempi_ignore_tkr -lmpi_mpifh -lmpi -lgfortran -lm > -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/9 -L/usr/lib/gcc/x86_64-linux-gnu/9 > -lgfortran -lm -lgcc_s -lquadmath -lpthread" > -DTPL_PARMETIS_INCLUDE_DIRS="/home/lgoodman/Projects/220826-vanilla-petsc-release-branch/installation/include;/usr/lib/x86_64-linux-gnu/openmpi/include;/usr/lib/x86_64-linux-gnu/openmpi/lib" > > -DTPL_PARMETIS_LIBRARIES="-Wl,-rpath,/home/lgoodman/Projects/220826-vanilla-petsc-release-branch/installation/lib > > -L/home/lgoodman/Projects/220826-vanilla-petsc-release-branch/installation/lib > -lparmetis > -Wl,-rpath,/home/lgoodman/Projects/220826-vanilla-petsc-release-branch/installation/lib > > -L/home/lgoodman/Projects/220826-vanilla-petsc-release-branch/installation/lib > -lmetis -lm -lcudart -lcufft -lcublas -lcusparse -lcusolver -lcurand -lcuda" > -DXSDK_INDEX_SIZE=64 -DXSDK_ENABLE_Fortran=ON -Denable_tests=0 > -Denable_examples=0 -DMPI_C_COMPILE_FLAGS:STRING="" > -DMPI_C_INCLUDE_PATH:STRING="" -DMPI_C_HEADER_DIR:STRING="" > -DMPI_C_LIBRARIES:STRING="" > > Or am I looking at the wrong thing? > > I also tried doing the build downloading petsc from the main branch but that > died at the same point and looked to give the same "redefinition of argument > 'optimize'" errors > > Thanks > Byron > > > > On Thu, Aug 25, 2022 at 4:15 PM Jacob Faibussowitsch <jacob....@gmail.com > <mailto:jacob....@gmail.com>> wrote: > > Sadly each one makes different -std=c++11 vs -std=c++14 choices as well. > > Main should hopefully *also* warn about this, but unless you really really > want -std=c++11 or -std=c++14, it is better to leave out this configure flag > (--with-cxx-dialect). > > If --with-cxx-dialect is unspecified (or set to “auto”) configure will pick > the most appropriate flag for you, taking your package environment into > account when choosing. If you specify the flag configure must assume the user > knows something it doesn’t and will blindly take what is given. > > Best regards, > > Jacob Faibussowitsch > (Jacob Fai - booss - oh - vitch) > > > On Aug 25, 2022, at 11:05, Barry Smith <bsm...@petsc.dev > > <mailto:bsm...@petsc.dev>> wrote: > > > > > > Oh yes, my mistake I misinterpreted the error message. It is actually the > > nvcc compiler complaining about multiple command line arguments related to > > optimization (which it is really picky about even if the two arguments are > > identical.) I reformatted some of the configure output to show this > > > > -ccbin mpicxx -std=c++14 -Xcompiler -fPIC -O3 -gencode > > arch=compute_75,code=sm_75 -Wno-deprecated-gpu-targets > > -I/usr/lib/x86_64-linux-gnu/openmpi/include/openmpi > > -I/usr/lib/x86_64-linux-gnu/openmpi/include > > -DDEBUGlevel=0 -DPRNTlevel=0 > > > > the next fragment is > > > > -O3 -DNDEBUG > > > > --generate-code=arch=compute_75,code=[compute_75,sm_75] > > > > -Xcompiler=-fPIC -std=c++11 -x cu -rdc=true -c > > /home/blahblah/Projects/220824-petsc-example-for_forum/petsc/arch-linux-c-opt/externalpackages/git.superlu_dist/SRC/superlu_gpu_utils.cu > > <http://superlu_gpu_utils.cu/> -o > > > > Hence the -O3 appears twice, once from the CMAKE argument PETSc passes to > > superlu_dist > > > > -DCMAKE_CUDA_FLAGS=" -ccbin mpicxx -std=c++14 -Xcompiler -fPIC -O3 -gencode > > arch=compute_75,code=sm_75 -Wno-deprecated-gpu-targets > > -I/usr/lib/x86_64-linux-gnu/openmpi/include/openmpi > > -I/usr/lib/x86_64-linux-gnu/openmpi/include -DDEBUGlevel=0 -DPRNTlevel=0" > > > > and then presumably, the superlu_dist cmake process adds its own arguments > > which overlap. > > > > If I remember correctly this is fixed in the main branch of PETSc, could > > you do an identical build with the main branch? If that works we will just > > need to figure out how to get MOOSE to work with main PETSc branch. > > > > Barry > > > > Sadly each one makes different -std=c++11 vs -std=c++14 choices as well. > > > > > > -------------------- > > > > [ 15%] Building CUDA object > > SRC/CMakeFiles/superlu_dist.dir/superlu_gpu_utils.cu.o > > cd > > /home/blahblah/Projects/220824-petsc-example-for_forum/petsc/arch-linux-c-opt/externalpackages/git.superlu_dist/petsc-build/SRC > > && /usr/bin/nvcc > > > > -DSUPERLU_DIST_EXPORTS -Dsuperlu_dist_EXPORTS > > > > -I/home/blahblah/Projects/220824-petsc-example-for_forum/petsc/arch-linux-c-opt/externalpackages/git.superlu_dist/petsc-build/SRC > > > > -I/home/blahblah/Projects/220824-petsc-build/installation/include > > -I/usr/lib/x86_64-linux-gnu/openmpi/lib > > > > -DUSE_VENDOR_BLAS -ccbin mpicxx -std=c++14 -Xcompiler -fPIC -O3 -gencode > > arch=compute_75,code=sm_75 -Wno-deprecated-gpu-targets > > > > -I/usr/lib/x86_64-linux-gnu/openmpi/include/openmpi > > -I/usr/lib/x86_64-linux-gnu/openmpi/include > > > > -DDEBUGlevel=0 -DPRNTlevel=0 -O3 -DNDEBUG > > > > --generate-code=arch=compute_75,code=[compute_75,sm_75] > > > > -Xcompiler=-fPIC -std=c++11 -x cu -rdc=true -c > > /home/blahblah/Projects/220824-petsc-example-for_forum/petsc/arch-linux-c-opt/externalpackages/git.superlu_dist/SRC/superlu_gpu_utils.cu > > <http://superlu_gpu_utils.cu/> -o > > > > CMakeFiles/superlu_dist.dir/superlu_gpu_utils.cu > > <http://superlu_gpu_utils.cu/> > > > > > > > >> On Aug 25, 2022, at 9:38 AM, byron <lbgpub...@gmail.com > >> <mailto:lbgpub...@gmail.com>> wrote: > >> > >> Hi Barry > >> > >> There doesn't appear to be any reference to 'optimize' in the only copy of > >> superlu_gpu_utils.cu <http://superlu_gpu_utils.cu/> I could find, I've > >> attached a copy. > >> > >> Thanks > >> > >> > >> > >> On Thu, Aug 25, 2022 at 1:44 PM Barry Smith <bsm...@petsc.dev > >> <mailto:bsm...@petsc.dev>> wrote: > >> > >> nvcc fatal : redefinition of argument 'optimize' > >> make[2]: *** [SRC/CMakeFiles/superlu_dist.dir/build.make:499: > >> SRC/CMakeFiles/superlu_dist.dir/superlu_gpu_utils.cu.o] Error 1 > >> make[2]: *** Waiting for unfinished jobs.... > >> nvcc fatal : redefinition of argument 'optimize' > >> make[2]: *** [SRC/CMakeFiles/superlu_dist-static.dir/build.make:499: > >> SRC/CMakeFiles/superlu_dist-static.dir/superlu_gpu_utils.cu.o] Error 1 > >> make[2]: *** Waiting for unfinished jobs.... > >> > >> It seems that nvcc is unhappy with something in superlu_gpu_utils.cu > >> <http://superlu_gpu_utils.cu/> Could you check that file for any > >> "redefinition of optimize"? Unfortunately the nvcc compiler isn't kind > >> enough to print the line number of the redefinition or that information is > >> lost somehow in the build process. > >> > >> Barry > >> > >>> On Aug 25, 2022, at 6:50 AM, byron <lbgpub...@gmail.com > >>> <mailto:lbgpub...@gmail.com>> wrote: > >>> > >>> Hi All > >>> > >>> I am having trouble getting petsc to build with cuda / superlu-dist > >>> enabled. I have been trying to build it as part of another software > >>> tool, moose, hence the choice of options being passed to configure > >>> > >>> python3 ./configure --download-hypre=1 \ > >>> --with-shared-libraries=1 \ > >>> --download-hdf5=1 \ > >>> --download-hdf5-fortran-bindings=0 \ > >>> --with-debugging=no \ > >>> --download-fblaslapack=1 \ > >>> --download-metis=1 \ > >>> --download-ptscotch=1 \ > >>> --download-parmetis=1 \ > >>> --download-superlu_dist=1 \ > >>> --download-mumps=1 \ > >>> --download-strumpack=1 \ > >>> --download-scalapack=1 \ > >>> --download-slepc=1 \ > >>> --with-mpi=1 \ > >>> --with-openmp=1 \ > >>> --with-cxx-dialect=C++11 \ > >>> --with-fortran-bindings=0 \ > >>> --with-sowing=0 \ > >>> --with-64-bit-indices \ > >>> --with-cuda \ > >>> --prefix=/home/blah/blah > >>> > >>> it fails with > >>> > >>> ============================================================================================= > >>> > >>> Installing PTScotch; this may take > >>> several minutes > >>> > >>> > >>> ============================================================================================= > >>> > >>> > >>> ============================================================================================= > >>> > >>> Trying to download > >>> git://https://github.com/xiaoyeli/superlu_dist > >>> <https://github.com/xiaoyeli/superlu_dist> for SUPERLU_DIST > >>> > >>> > >>> ============================================================================================= > >>> > >>> > >>> ============================================================================================= > >>> > >>> Configuring SUPERLU_DIST with cmake; > >>> this may take several minutes > >>> > >>> > >>> ============================================================================================= > >>> > >>> > >>> ============================================================================================= > >>> > >>> Compiling and installing SUPERLU_DIST; > >>> this may take several minutes > >>> > >>> > >>> ============================================================================================= > >>> > >>> > >>> ******************************************************************************* > >>> UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log for > >>> details): > >>> ------------------------------------------------------------------------------- > >>> Error running make on SUPERLU_DIST > >>> ******************************************************************************* > >>> > >>> I've attached the configure.log. > >>> > >>> I've tried a few different things but nothing that resulted in a > >>> successfull build. The most promising was I tried to build superlu_dist > >>> separately with the configure options that petsc passes it and then kept > >>> trying the build over and over droping out different configure options > >>> until I got it to work. In that case removing a "-std=c++14" entry > >>> appeared to fix the problem but then when I made the same change in how > >>> petsc builds superlu-dist the make failed on superlu-dist with a > >>> different error. I get the feeling I'm dealing with symptoms of the > >>> problem rather than the cause. > >>> > >>> Thanks > >>> Byron > >>> > >>> > >>> > >>> > >>> <configure.log> > >> > >> <superlu_gpu_utils.cu <http://superlu_gpu_utils.cu/>> > > >
