It looks like I've (we've) cracked it, at least the petsc stage of what I'm trying to do. To get the configure to work I used the following
python3 ./configure --download-hypre=1 --with-shared-libraries=1 --download-hdf5=1 --download-hdf5-fortran-bindings=0 --with-debugging=no --download-fblaslapack=1 --download-metis=1 --download-ptscotch=1 --download-parmetis=1 --download-superlu_dist=1 --download-mumps=1 --download-strumpack=1 --download-scalapack=1 --download-slepc=1 --with-mpi=1 --with-openmp=1 --with-cxx-dialect=C++11 -with-fortran-bindings=0 --with-sowing=0 --with-64-bit-indices --with-cuda -CUDAOPTFLAGS="" --with-cuda-dialect=cxx11 --prefix=/home/lgoodman/Projects/petscExamples/main-build-CUDAOPTFLAGS-zero-cxx-dialect-unspc-cuda-dial-11/installation Where -CUDAOPTFLAGS="" was the solutions to getting rid of the duplicate -O3 then I was getting errors about 'redefinition of argument 'std'" so i added --with-cxx-dialect=C++11 and --with-cuda-dialect=cxx11 and that got rid of those errors. I image there are cleaner / more elegant ways of doing this but I'll settle for this for now. Thanks a lot for the help guys, much appreciated Best regards Byron On Fri, Aug 26, 2022 at 4:06 PM Barry Smith <bsm...@petsc.dev> wrote: > > Yes, PETSc sends down a single -O3 to SuperLU_DIST CMake but internally > SuperLU_DIST CMake is adding its own set of nvcc compiler flags to use and > these that get added to the one's PETSc passes in (they do not replace > them). A little further down in the file it displays a compile line with > nvcc that has two sets of -O3 (as well as two sets of other things as well). > > I think we changed the main branch PETSc to not pass down the > optimization flags to the CMake CUDA to prevent the problem from happening. > In other words, we let the package decide its optimization flags instead of > trying to tell it what to use. > > Barry > > The root of the problem is that compiler APIs are a bunch of strings > passed on the command line to the compiler without any structure. > > > On Aug 26, 2022, at 9:47 AM, byron <lbgpub...@gmail.com> wrote: > > Hi Barry > > I dont think I see the same thing. > > When I do a build using the release branch petsc passes the following > arguements to superlu_dist (taken from > /home/blahblah/Projects/220826-vanilla-petsc-release-branch/petsc/arch-linux-c-opt/externalpackages/git.superlu_dist/superlu_dist.petscconf) > which has only one -O3 entry > > -DCMAKE_INSTALL_PREFIX=/home/blahblah/Projects/220826-vanilla-petsc-release-branch/installation > -DCMAKE_INSTALL_NAME_DIR:STRING="/home/lgoodman/Projects/220826-vanilla-petsc-release-branch/installation/lib" > -DCMAKE_INSTALL_LIBDIR:STRING="lib" -DCMAKE_VERBOSE_MAKEFILE=1 > -DCMAKE_BUILD_TYPE=Release -DCMAKE_C_COMPILER="mpicc" > -DMPI_C_COMPILER="/usr/bin/mpicc" -DCMAKE_AR=/usr/bin/ar > -DCMAKE_RANLIB=/usr/bin/ranlib -DCMAKE_C_FLAGS:STRING="-fPIC > -Wno-lto-type-mismatch -fstack-protector -g -O -fopenmp -DDEBUGlevel=0 > -DPRNTlevel=0" -DCMAKE_C_FLAGS_DEBUG:STRING="-fPIC -Wno-lto-type-mismatch > -fstack-protector -g -O -fopenmp -DDEBUGlevel=0 -DPRNTlevel=0" > -DCMAKE_C_FLAGS_RELEASE:STRING="-fPIC -Wno-lto-type-mismatch > -fstack-protector -g -O -fopenmp -DDEBUGlevel=0 -DPRNTlevel=0" > -DCMAKE_CXX_COMPILER="mpicxx" -DMPI_CXX_COMPILER="/usr/bin/mpicxx" > -DCMAKE_CXX_FLAGS:STRING="-Wno-lto-type-mismatch -fstack-protector -g -O > -fopenmp -std=c++11 -fPIC -fopenmp -DDEBUGlevel=0 -DPRNTlevel=0" > -DCMAKE_CXX_FLAGS_DEBUG:STRING="-Wno-lto-type-mismatch -fstack-protector -g > -O -fopenmp -std=c++11 -fPIC -fopenmp -DDEBUGlevel=0 -DPRNTlevel=0" > -DCMAKE_CXX_FLAGS_RELEASE:STRING="-Wno-lto-type-mismatch -fstack-protector > -g -O -fopenmp -std=c++11 -fPIC -fopenmp -DDEBUGlevel=0 -DPRNTlevel=0" > -DCMAKE_Fortran_COMPILER="mpif90" -DMPI_Fortran_COMPILER="/usr/bin/mpif90" > -DCMAKE_Fortran_FLAGS:STRING="-fPIC -ffree-line-length-0 > -Wno-lto-type-mismatch -g -O -fopenmp" > -DCMAKE_Fortran_FLAGS_DEBUG:STRING="-fPIC -ffree-line-length-0 > -Wno-lto-type-mismatch -g -O -fopenmp" > -DCMAKE_Fortran_FLAGS_RELEASE:STRING="-fPIC -ffree-line-length-0 > -Wno-lto-type-mismatch -g -O -fopenmp" -DCMAKE_EXE_LINKER_FLAGS:STRING=" > -fopenmp -fopenmp" -DBUILD_SHARED_LIBS:BOOL=ON -DTPL_ENABLE_CUDALIB=TRUE > -DTPL_CUDA_LIBRARIES="-lcudart -lcufft -lcublas -lcusparse -lcusolver > -lcurand -lcuda" -DCUDA_ARCH_FLAGS="-arch=sm_75" > -DCMAKE_CUDA_COMPILER="nvcc" -DCMAKE_CUDA_FLAGS=" -ccbin mpicxx -std=c++14 > -Xcompiler -fPIC *-O3 *-gencode arch=compute_75,code=sm_75 > -Wno-deprecated-gpu-targets > -I/usr/lib/x86_64-linux-gnu/openmpi/include/openmpi > -I/usr/lib/x86_64-linux-gnu/openmpi/include -DDEBUGlevel=0 -DPRNTlevel=0" > -DUSE_XSDK_DEFAULTS=YES > -DTPL_BLAS_LIBRARIES="-Wl,-rpath,/home/lgoodman/Projects/220826-vanilla-petsc-release-branch/installation/lib > -L/home/lgoodman/Projects/220826-vanilla-petsc-release-branch/installation/lib > -lflapack > -Wl,-rpath,/home/lgoodman/Projects/220826-vanilla-petsc-release-branch/installation/lib > -L/home/lgoodman/Projects/220826-vanilla-petsc-release-branch/installation/lib > -lfblas -lcudart -lcufft -lcublas -lcusparse -lcusolver -lcurand -lcuda -lm > -lstdc++ -ldl -Wl,-rpath,/usr/lib/x86_64-linux-gnu/openmpi/lib > -L/usr/lib/x86_64-linux-gnu/openmpi/lib -lmpi_usempif08 > -lmpi_usempi_ignore_tkr -lmpi_mpifh -lmpi -lgfortran -lm > -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/9 > -L/usr/lib/gcc/x86_64-linux-gnu/9 -lgfortran -lm -lgcc_s -lquadmath > -lpthread" > -DTPL_LAPACK_LIBRARIES="-Wl,-rpath,/home/lgoodman/Projects/220826-vanilla-petsc-release-branch/installation/lib > -L/home/lgoodman/Projects/220826-vanilla-petsc-release-branch/installation/lib > -lflapack > -Wl,-rpath,/home/lgoodman/Projects/220826-vanilla-petsc-release-branch/installation/lib > -L/home/lgoodman/Projects/220826-vanilla-petsc-release-branch/installation/lib > -lfblas -lcudart -lcufft -lcublas -lcusparse -lcusolver -lcurand -lcuda -lm > -lstdc++ -ldl -Wl,-rpath,/usr/lib/x86_64-linux-gnu/openmpi/lib > -L/usr/lib/x86_64-linux-gnu/openmpi/lib -lmpi_usempif08 > -lmpi_usempi_ignore_tkr -lmpi_mpifh -lmpi -lgfortran -lm > -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/9 > -L/usr/lib/gcc/x86_64-linux-gnu/9 -lgfortran -lm -lgcc_s -lquadmath > -lpthread" > -DTPL_PARMETIS_INCLUDE_DIRS="/home/lgoodman/Projects/220826-vanilla-petsc-release-branch/installation/include;/usr/lib/x86_64-linux-gnu/openmpi/include;/usr/lib/x86_64-linux-gnu/openmpi/lib" > -DTPL_PARMETIS_LIBRARIES="-Wl,-rpath,/home/lgoodman/Projects/220826-vanilla-petsc-release-branch/installation/lib > -L/home/lgoodman/Projects/220826-vanilla-petsc-release-branch/installation/lib > -lparmetis > -Wl,-rpath,/home/lgoodman/Projects/220826-vanilla-petsc-release-branch/installation/lib > -L/home/lgoodman/Projects/220826-vanilla-petsc-release-branch/installation/lib > -lmetis -lm -lcudart -lcufft -lcublas -lcusparse -lcusolver -lcurand > -lcuda" -DXSDK_INDEX_SIZE=64 -DXSDK_ENABLE_Fortran=ON -Denable_tests=0 > -Denable_examples=0 -DMPI_C_COMPILE_FLAGS:STRING="" > -DMPI_C_INCLUDE_PATH:STRING="" -DMPI_C_HEADER_DIR:STRING="" > -DMPI_C_LIBRARIES:STRING="" > > Or am I looking at the wrong thing? > > I also tried doing the build downloading petsc from the main branch but > that died at the same point and looked to give the same "redefinition of > argument 'optimize'" errors > > Thanks > Byron > > > > On Thu, Aug 25, 2022 at 4:15 PM Jacob Faibussowitsch <jacob....@gmail.com> > wrote: > >> > Sadly each one makes different -std=c++11 vs -std=c++14 choices as >> well. >> >> Main should hopefully *also* warn about this, but unless you really >> really want -std=c++11 or -std=c++14, it is better to leave out this >> configure flag (--with-cxx-dialect). >> >> If --with-cxx-dialect is unspecified (or set to “auto”) configure will >> pick the most appropriate flag for you, taking your package environment >> into account when choosing. If you specify the flag configure must assume >> the user knows something it doesn’t and will blindly take what is given. >> >> Best regards, >> >> Jacob Faibussowitsch >> (Jacob Fai - booss - oh - vitch) >> >> > On Aug 25, 2022, at 11:05, Barry Smith <bsm...@petsc.dev> wrote: >> > >> > >> > Oh yes, my mistake I misinterpreted the error message. It is actually >> the nvcc compiler complaining about multiple command line arguments related >> to optimization (which it is really picky about even if the two arguments >> are identical.) I reformatted some of the configure output to show this >> > >> > -ccbin mpicxx -std=c++14 -Xcompiler -fPIC -O3 -gencode >> arch=compute_75,code=sm_75 -Wno-deprecated-gpu-targets >> > -I/usr/lib/x86_64-linux-gnu/openmpi/include/openmpi >> -I/usr/lib/x86_64-linux-gnu/openmpi/include >> > -DDEBUGlevel=0 -DPRNTlevel=0 >> > >> > the next fragment is >> > >> > -O3 -DNDEBUG >> > >> > --generate-code=arch=compute_75,code=[compute_75,sm_75] >> > >> > -Xcompiler=-fPIC -std=c++11 -x cu -rdc=true -c >> /home/blahblah/Projects/220824-petsc-example-for_forum/petsc/arch-linux-c-opt/externalpackages/git.superlu_dist/SRC/ >> superlu_gpu_utils.cu -o >> > >> > Hence the -O3 appears twice, once from the CMAKE argument PETSc passes >> to superlu_dist >> > >> > -DCMAKE_CUDA_FLAGS=" -ccbin mpicxx -std=c++14 -Xcompiler -fPIC -O3 >> -gencode arch=compute_75,code=sm_75 -Wno-deprecated-gpu-targets >> -I/usr/lib/x86_64-linux-gnu/openmpi/include/openmpi >> -I/usr/lib/x86_64-linux-gnu/openmpi/include -DDEBUGlevel=0 -DPRNTlevel=0" >> > >> > and then presumably, the superlu_dist cmake process adds its own >> arguments which overlap. >> > >> > If I remember correctly this is fixed in the main branch of PETSc, >> could you do an identical build with the main branch? If that works we will >> just need to figure out how to get MOOSE to work with main PETSc branch. >> > >> > Barry >> > >> > Sadly each one makes different -std=c++11 vs -std=c++14 choices as >> well. >> > >> > >> > -------------------- >> > >> > [ 15%] Building CUDA object >> SRC/CMakeFiles/superlu_dist.dir/superlu_gpu_utils.cu.o >> > cd >> /home/blahblah/Projects/220824-petsc-example-for_forum/petsc/arch-linux-c-opt/externalpackages/git.superlu_dist/petsc-build/SRC >> && /usr/bin/nvcc >> > >> > -DSUPERLU_DIST_EXPORTS -Dsuperlu_dist_EXPORTS >> > >> > >> -I/home/blahblah/Projects/220824-petsc-example-for_forum/petsc/arch-linux-c-opt/externalpackages/git.superlu_dist/petsc-build/SRC >> > >> > -I/home/blahblah/Projects/220824-petsc-build/installation/include >> -I/usr/lib/x86_64-linux-gnu/openmpi/lib >> > >> > -DUSE_VENDOR_BLAS -ccbin mpicxx -std=c++14 -Xcompiler -fPIC -O3 >> -gencode arch=compute_75,code=sm_75 -Wno-deprecated-gpu-targets >> > >> > -I/usr/lib/x86_64-linux-gnu/openmpi/include/openmpi >> -I/usr/lib/x86_64-linux-gnu/openmpi/include >> > >> > -DDEBUGlevel=0 -DPRNTlevel=0 -O3 -DNDEBUG >> > >> > --generate-code=arch=compute_75,code=[compute_75,sm_75] >> > >> > -Xcompiler=-fPIC -std=c++11 -x cu -rdc=true -c >> /home/blahblah/Projects/220824-petsc-example-for_forum/petsc/arch-linux-c-opt/externalpackages/git.superlu_dist/SRC/ >> superlu_gpu_utils.cu -o >> > >> > CMakeFiles/superlu_dist.dir/superlu_gpu_utils.cu >> > >> > >> > >> >> On Aug 25, 2022, at 9:38 AM, byron <lbgpub...@gmail.com> wrote: >> >> >> >> Hi Barry >> >> >> >> There doesn't appear to be any reference to 'optimize' in the only >> copy of superlu_gpu_utils.cu I could find, I've attached a copy. >> >> >> >> Thanks >> >> >> >> >> >> >> >> On Thu, Aug 25, 2022 at 1:44 PM Barry Smith <bsm...@petsc.dev> wrote: >> >> >> >> nvcc fatal : redefinition of argument 'optimize' >> >> make[2]: *** [SRC/CMakeFiles/superlu_dist.dir/build.make:499: >> SRC/CMakeFiles/superlu_dist.dir/superlu_gpu_utils.cu.o] Error 1 >> >> make[2]: *** Waiting for unfinished jobs.... >> >> nvcc fatal : redefinition of argument 'optimize' >> >> make[2]: *** [SRC/CMakeFiles/superlu_dist-static.dir/build.make:499: >> SRC/CMakeFiles/superlu_dist-static.dir/superlu_gpu_utils.cu.o] Error 1 >> >> make[2]: *** Waiting for unfinished jobs.... >> >> >> >> It seems that nvcc is unhappy with something in superlu_gpu_utils.cu >> Could you check that file for any "redefinition of optimize"? Unfortunately >> the nvcc compiler isn't kind enough to print the line number of the >> redefinition or that information is lost somehow in the build process. >> >> >> >> Barry >> >> >> >>> On Aug 25, 2022, at 6:50 AM, byron <lbgpub...@gmail.com> wrote: >> >>> >> >>> Hi All >> >>> >> >>> I am having trouble getting petsc to build with cuda / superlu-dist >> enabled. I have been trying to build it as part of another software tool, >> moose, hence the choice of options being passed to configure >> >>> >> >>> python3 ./configure --download-hypre=1 \ >> >>> --with-shared-libraries=1 \ >> >>> --download-hdf5=1 \ >> >>> --download-hdf5-fortran-bindings=0 \ >> >>> --with-debugging=no \ >> >>> --download-fblaslapack=1 \ >> >>> --download-metis=1 \ >> >>> --download-ptscotch=1 \ >> >>> --download-parmetis=1 \ >> >>> --download-superlu_dist=1 \ >> >>> --download-mumps=1 \ >> >>> --download-strumpack=1 \ >> >>> --download-scalapack=1 \ >> >>> --download-slepc=1 \ >> >>> --with-mpi=1 \ >> >>> --with-openmp=1 \ >> >>> --with-cxx-dialect=C++11 \ >> >>> --with-fortran-bindings=0 \ >> >>> --with-sowing=0 \ >> >>> --with-64-bit-indices \ >> >>> --with-cuda \ >> >>> --prefix=/home/blah/blah >> >>> >> >>> it fails with >> >>> >> >>> >> ============================================================================================= >> >> Installing PTScotch; this may take several >> minutes >> >> ============================================================================================= >> >> >> >> ============================================================================================= >> >> Trying to download git:// >> https://github.com/xiaoyeli/superlu_dist for SUPERLU_DIST >> >> >> ============================================================================================= >> >> >> >> ============================================================================================= >> >> Configuring SUPERLU_DIST with cmake; this >> may take several minutes >> >> ============================================================================================= >> >> >> >> ============================================================================================= >> >> Compiling and installing SUPERLU_DIST; this >> may take several minutes >> >> ============================================================================================= >> >> >> >> ******************************************************************************* >> >>> UNABLE to CONFIGURE with GIVEN OPTIONS (see configure.log >> for details): >> >>> >> ------------------------------------------------------------------------------- >> >>> Error running make on SUPERLU_DIST >> >>> >> ******************************************************************************* >> >>> >> >>> I've attached the configure.log. >> >>> >> >>> I've tried a few different things but nothing that resulted in a >> successfull build. The most promising was I tried to build superlu_dist >> separately with the configure options that petsc passes it and then kept >> trying the build over and over droping out different configure options >> until I got it to work. In that case removing a "-std=c++14" entry >> appeared to fix the problem but then when I made the same change in how >> petsc builds superlu-dist the make failed on superlu-dist with a different >> error. I get the feeling I'm dealing with symptoms of the problem rather >> than the cause. >> >>> >> >>> Thanks >> >>> Byron >> >>> >> >>> >> >>> >> >>> >> >>> <configure.log> >> >> >> >> <superlu_gpu_utils.cu> >> > >> >> >
