On Thu, Mar 15, 2018 at 8:18 AM, Manuel Valera <mvaler...@mail.sdsu.edu> wrote:
> Ok so, i went back and erased the old libpetsc.so.3 i think it was the one > causing problems, i had --with-shared-libraries=0 and the installation > complained of not having that file, then reinstalled with > --with-shared-libraries=1 and it is finally recognizing my system > installation with only CUDA, but now it gives a 11 SEGV violation error: > But ex19 runs? And this crash is without CUDA? Just run valgrind on it Matt > [valera@node50 alone]$ ./linsolve > laplacian.petsc ! > TrivSoln loaded, size: 125 / 125 > [0]PETSC ERROR: ------------------------------ > ------------------------------------------ > [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, > probably memory access out of range > [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/ > documentation/faq.html#valgrind > [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS > X to find memory corruption errors > [0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and > run > [0]PETSC ERROR: to get more information on the crash. > [0]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [0]PETSC ERROR: Signal received > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > [0]PETSC ERROR: Petsc Development GIT revision: v3.8.3-2027-g045eeab GIT > Date: 2018-03-12 13:30:25 -0500 > [0]PETSC ERROR: ./linsolve on a named node50 by valera Wed Mar 14 > 16:17:34 2018 > [0]PETSC ERROR: Configure options --prefix=/usr/local/petsc.cod/petsc-install > --with-mpi-dir=/usr/lib64/openmpi --with-blaslapack-dir=/usr/lib64 > COPTFLAGS=-O2 CXXOPTFLAGS=-O2 FOPTFLAGS=-O2 --with-shared-libraries=1 > --with-debugging=0 --with-cuda=1 --with-cuda-arch=sm_60 > [0]PETSC ERROR: #1 User provided function() line 0 in unknown file > -------------------------------------------------------------------------- > MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD > with errorcode 59. > > NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. > You may or may not see output from other processes, depending on > exactly when Open MPI kills them. > -------------------------------------------------------------------------- > [valera@node50 alone]$ > > > > > Thanks, > > > > > > > > On Wed, Mar 14, 2018 at 1:52 PM, Matthew Knepley <knep...@gmail.com> > wrote: > >> On Thu, Mar 15, 2018 at 4:01 AM, Manuel Valera <mvaler...@mail.sdsu.edu> >> wrote: >> >>> Ok well, it turns out the $PETSC_DIR points to the testpetsc directory, >>> and it makes, install and tests without problems (only a problem on ex5f) >>> but trying to reconfigure on valera/petsc directory asks me to change the >>> $PETSC_DIR variable, >>> >> >> Yes, you must set PETSC_DIR to point to the installation you want to use. >> >> Matt >> >> >>> Meanwhile the system installation still points to the valera/petsc/cuda >>> build, >>> >>> should i just delete the petsc installation folder and start over? >>> >>> Thanks, >>> >>> On Wed, Mar 14, 2018 at 11:36 AM, Matthew Knepley <knep...@gmail.com> >>> wrote: >>> >>>> On Thu, Mar 15, 2018 at 3:25 AM, Manuel Valera <mvaler...@mail.sdsu.edu >>>> > wrote: >>>> >>>>> yeah that worked, >>>>> >>>>> [valera@node50 tutorials]$ ./ex19 -dm_vec_type seqcuda -dm_mat_type >>>>> seqaijcusparse >>>>> lid velocity = 0.0625, prandtl # = 1., grashof # = 1. >>>>> Number of SNES iterations = 2 >>>>> [valera@node50 tutorials]$ >>>>> >>>>> How do i make sure the other program refer to this installation? using >>>>> the same arguments there i get: >>>>> >>>>> [valera@node50 alone]$ ./linsolve -vec_type seqcuda -mat_type >>>>> seqaijcusparse >>>>> laplacian.petsc ! >>>>> [0]PETSC ERROR: --------------------- Error Message >>>>> -------------------------------------------------------------- >>>>> [0]PETSC ERROR: Unknown type. Check for miss-spelling or missing >>>>> package: http://www.mcs.anl.gov/petsc/documentation/installation.html >>>>> #external >>>>> [0]PETSC ERROR: Unknown vector type: seqcuda >>>>> >>>> >>>> This PETSc has not been configured with CUDA. It is located in >>>> /home/valera/petsc. >>>> The other >>>> one you used is located in /home/valera/testpetsc. It does not make >>>> much sense to me that >>>> it does not understand CUDA since it says the configure arguments had >>>> --with-cuda=1. There >>>> must have been a build problem. Rebuild >>>> >>>> cd $PETSC_DIR >>>> make all >>>> >>>> If you still have a problem, reconfigure >>>> >>>> cd $PETSC_DIR >>>> ./cuda/lib/petsc/conf/reconfigure-cuda.py >>>> make all >>>> >>>> If that still fails, then something very bizarre is happening on your >>>> machine and we will have >>>> to exchange more mail. >>>> >>>> Matt >>>> >>>> >>>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/d >>>>> ocumentation/faq.html for trouble shooting. >>>>> [0]PETSC ERROR: Petsc Development GIT revision: v3.8.3-1817-g96b6f8a >>>>> GIT Date: 2018-02-28 10:19:08 -0600 >>>>> [0]PETSC ERROR: ./linsolve on a cuda named node50 by valera Wed Mar 14 >>>>> 11:25:11 2018 >>>>> [0]PETSC ERROR: Configure options PETSC_ARCH=cuda --with-cc=mpicc >>>>> --with-cxx=mpic++ --with-fc=mpifort --COPTFLAGS=-O3 --CXXOPTFLAGS=-O3 >>>>> --FOPTFLAGS=-O3 --with-shared-libraries=1 --with-debugging=1 --with-cuda=1 >>>>> --with-cuda-arch=sm_60 --with-cusp=1 --with-cusp-dir=/home/valera/cusp >>>>> --with-vienacl=1 --download-fblaslapack=1 --download-hypre >>>>> [0]PETSC ERROR: #1 VecSetType() line 42 in >>>>> /home/valera/petsc/src/vec/vec/interface/vecreg.c >>>>> [0]PETSC ERROR: #2 VecSetTypeFromOptions_Private() line 1241 in >>>>> /home/valera/petsc/src/vec/vec/interface/vector.c >>>>> [0]PETSC ERROR: #3 VecSetFromOptions() line 1276 in >>>>> /home/valera/petsc/src/vec/vec/interface/vector.c >>>>> [0]PETSC ERROR: --------------------- Error Message >>>>> -------------------------------------------------------------- >>>>> [0]PETSC ERROR: Object is in wrong state >>>>> [0]PETSC ERROR: Vec object's type is not set: Argument # 1 >>>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/d >>>>> ocumentation/faq.html for trouble shooting. >>>>> [0]PETSC ERROR: Petsc Development GIT revision: v3.8.3-1817-g96b6f8a >>>>> GIT Date: 2018-02-28 10:19:08 -0600 >>>>> [0]PETSC ERROR: ./linsolve on a cuda named node50 by valera Wed Mar 14 >>>>> 11:25:11 2018 >>>>> [0]PETSC ERROR: Configure options PETSC_ARCH=cuda --with-cc=mpicc >>>>> --with-cxx=mpic++ --with-fc=mpifort --COPTFLAGS=-O3 --CXXOPTFLAGS=-O3 >>>>> --FOPTFLAGS=-O3 --with-shared-libraries=1 --with-debugging=1 --with-cuda=1 >>>>> --with-cuda-arch=sm_60 --with-cusp=1 --with-cusp-dir=/home/valera/cusp >>>>> --with-vienacl=1 --download-fblaslapack=1 --download-hypre >>>>> [0]PETSC ERROR: #4 VecDuplicate() line 375 in >>>>> /home/valera/petsc/src/vec/vec/interface/vector.c >>>>> [0]PETSC ERROR: --------------------- Error Message >>>>> -------------------------------------------------------------- >>>>> [0]PETSC ERROR: Object is in wrong state >>>>> [0]PETSC ERROR: Vec object's type is not set: Argument # 1 >>>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/d >>>>> ocumentation/faq.html for trouble shooting. >>>>> [0]PETSC ERROR: Petsc Development GIT revision: v3.8.3-1817-g96b6f8a >>>>> GIT Date: 2018-02-28 10:19:08 -0600 >>>>> [0]PETSC ERROR: ./linsolve on a cuda named node50 by valera Wed Mar 14 >>>>> 11:25:11 2018 >>>>> [0]PETSC ERROR: Configure options PETSC_ARCH=cuda --with-cc=mpicc >>>>> --with-cxx=mpic++ --with-fc=mpifort --COPTFLAGS=-O3 --CXXOPTFLAGS=-O3 >>>>> --FOPTFLAGS=-O3 --with-shared-libraries=1 --with-debugging=1 --with-cuda=1 >>>>> --with-cuda-arch=sm_60 --with-cusp=1 --with-cusp-dir=/home/valera/cusp >>>>> --with-vienacl=1 --download-fblaslapack=1 --download-hypre >>>>> [0]PETSC ERROR: #5 VecDuplicate() line 375 in >>>>> /home/valera/petsc/src/vec/vec/interface/vector.c >>>>> [0]PETSC ERROR: #6 User provided function() line 0 in User file >>>>> [valera@node50 alone]$ >>>>> >>>>> I made sure there is a call for Vec/MatSetFromOptions() there, i am >>>>> loading the matrix from a petsc binary in this case, >>>>> >>>>> Thanks, >>>>> >>>>> >>>>> On Wed, Mar 14, 2018 at 11:22 AM, Matthew Knepley <knep...@gmail.com> >>>>> wrote: >>>>> >>>>>> On Thu, Mar 15, 2018 at 3:19 AM, Manuel Valera < >>>>>> mvaler...@mail.sdsu.edu> wrote: >>>>>> >>>>>>> Yes, this is the system installation that is being correctly linked >>>>>>> (the linear solver and model are not linking the correct installation >>>>>>> idk >>>>>>> why yet) i configured with only CUDA this time because of the message >>>>>>> Karl >>>>>>> Rupp posted on my installation thread, where he says only one type of >>>>>>> library will work at a time, anyway this is what i got: >>>>>>> >>>>>>> [valera@node50 tutorials]$ ./ex19 -dm_vec_type seqcuda -dm_mat_type >>>>>>> seqaijcusp >>>>>>> lid velocity = 0.0625, prandtl # = 1., grashof # = 1. >>>>>>> [0]PETSC ERROR: --------------------- Error Message >>>>>>> -------------------------------------------------------------- >>>>>>> [0]PETSC ERROR: Unknown type. Check for miss-spelling or missing >>>>>>> package: http://www.mcs.anl.gov/petsc/d >>>>>>> ocumentation/installation.html#external >>>>>>> [0]PETSC ERROR: Unknown Mat type given: seqaijcusp >>>>>>> >>>>>> >>>>>> It is telling you the problem. Use >>>>>> >>>>>> -dm_mat_type seqaijcusparse >>>>>> >>>>>> Matt >>>>>> >>>>>> >>>>>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/d >>>>>>> ocumentation/faq.html for trouble shooting. >>>>>>> [0]PETSC ERROR: Petsc Development GIT revision: >>>>>>> v3.8.3-2027-g045eeab GIT Date: 2018-03-12 13:30:25 -0500 >>>>>>> [0]PETSC ERROR: ./ex19 on a named node50 by valera Wed Mar 14 >>>>>>> 11:17:25 2018 >>>>>>> [0]PETSC ERROR: Configure options >>>>>>> --prefix=/usr/local/petsc.cod/petsc-install >>>>>>> --with-mpi-dir=/usr/lib64/openmpi --with-blaslapack-dir=/usr/lib64 >>>>>>> COPTFLAGS=-O2 CXXOPTFLAGS=-O2 FOPTFLAGS=-O2 --with-shared-libraries=0 >>>>>>> --download-hypre --with-debugging=0 --with-cuda=1 --with-cuda-arch=sm_60 >>>>>>> [0]PETSC ERROR: #1 MatSetType() line 61 in >>>>>>> /home/valera/testpetsc/src/mat/interface/matreg.c >>>>>>> [0]PETSC ERROR: #2 DMCreateMatrix_DA() line 693 in >>>>>>> /home/valera/testpetsc/src/dm/impls/da/fdda.c >>>>>>> [0]PETSC ERROR: #3 DMCreateMatrix() line 1199 in >>>>>>> /home/valera/testpetsc/src/dm/interface/dm.c >>>>>>> [0]PETSC ERROR: #4 SNESSetUpMatrices() line 646 in >>>>>>> /home/valera/testpetsc/src/snes/interface/snes.c >>>>>>> [0]PETSC ERROR: #5 SNESSetUp_NEWTONLS() line 296 in >>>>>>> /home/valera/testpetsc/src/snes/impls/ls/ls.c >>>>>>> [0]PETSC ERROR: #6 SNESSetUp() line 2795 in >>>>>>> /home/valera/testpetsc/src/snes/interface/snes.c >>>>>>> [0]PETSC ERROR: #7 SNESSolve() line 4187 in >>>>>>> /home/valera/testpetsc/src/snes/interface/snes.c >>>>>>> [0]PETSC ERROR: #8 main() line 161 in /home/valera/testpetsc/src/sne >>>>>>> s/examples/tutorials/ex19.c >>>>>>> [0]PETSC ERROR: PETSc Option Table entries: >>>>>>> [0]PETSC ERROR: -dm_mat_type seqaijcusp >>>>>>> [0]PETSC ERROR: -dm_vec_type seqcuda >>>>>>> [0]PETSC ERROR: ----------------End of Error Message -------send >>>>>>> entire error message to petsc-ma...@mcs.anl.gov---------- >>>>>>> ------------------------------------------------------------ >>>>>>> -------------- >>>>>>> MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD >>>>>>> with errorcode 86. >>>>>>> >>>>>>> NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. >>>>>>> You may or may not see output from other processes, depending on >>>>>>> exactly when Open MPI kills them. >>>>>>> ------------------------------------------------------------ >>>>>>> -------------- >>>>>>> >>>>>>> >>>>>>> On Wed, Mar 14, 2018 at 11:16 AM, Matthew Knepley <knep...@gmail.com >>>>>>> > wrote: >>>>>>> >>>>>>>> On Thu, Mar 15, 2018 at 3:12 AM, Manuel Valera < >>>>>>>> mvaler...@mail.sdsu.edu> wrote: >>>>>>>> >>>>>>>>> Thanks, got this error: >>>>>>>>> >>>>>>>> >>>>>>>> Did you not configure with CUSP? It looks like you have CUDA, so use >>>>>>>> >>>>>>>> -dm_vec_type seqcuda >>>>>>>> >>>>>>>> Thanks, >>>>>>>> >>>>>>>> Matt >>>>>>>> >>>>>>>> >>>>>>>>> [valera@node50 testpetsc]$ cd src/snes/examples/tutorials/ >>>>>>>>> [valera@node50 tutorials]$ PETSC_ARCH="" make ex19 >>>>>>>>> /usr/lib64/openmpi/bin/mpicc -o ex19.o -c -Wall -Wwrite-strings >>>>>>>>> -Wno-strict-aliasing -Wno-unknown-pragmas -fstack-protector >>>>>>>>> -fvisibility=hidden -O2 -I/home/valera/testpetsc/include >>>>>>>>> -I/home/valera/testpetsc/arch-linux2-c-opt/include >>>>>>>>> -I/usr/local/petsc.cod/petsc-install/include >>>>>>>>> -I/usr/local/cuda/include -I/usr/lib64/openmpi/include `pwd`/ex19.c >>>>>>>>> /usr/lib64/openmpi/bin/mpicc -Wall -Wwrite-strings >>>>>>>>> -Wno-strict-aliasing -Wno-unknown-pragmas -fstack-protector >>>>>>>>> -fvisibility=hidden -O2 -o ex19 ex19.o >>>>>>>>> -L/home/valera/testpetsc/arch-linux2-c-opt/lib >>>>>>>>> -Wl,-rpath,/usr/local/petsc.cod/petsc-install/lib >>>>>>>>> -L/usr/local/petsc.cod/petsc-install/lib -Wl,-rpath,/usr/lib64 >>>>>>>>> -L/usr/lib64 -Wl,-rpath,/usr/local/cuda/lib64 >>>>>>>>> -L/usr/local/cuda/lib64 -L/usr/lib64/openmpi/lib >>>>>>>>> -L/usr/lib/gcc/x86_64-redhat-linux/4.8.5 >>>>>>>>> -Wl,-rpath,/usr/lib64/openmpi/lib -lpetsc -lHYPRE -llapack -lblas >>>>>>>>> -lm -lcufft -lcublas -lcudart -lcusparse -lX11 -lstdc++ -ldl >>>>>>>>> -lmpi_usempi >>>>>>>>> -lmpi_mpifh -lmpi -lgfortran -lm -lgfortran -lm -lgcc_s -lquadmath >>>>>>>>> -lpthread -lstdc++ -ldl >>>>>>>>> /usr/bin/rm -f ex19.o >>>>>>>>> [valera@node50 tutorials]$ ./ex19 -dm_vec_type seqcusp >>>>>>>>> -dm_mat_type seqaijcusp >>>>>>>>> lid velocity = 0.0625, prandtl # = 1., grashof # = 1. >>>>>>>>> [0]PETSC ERROR: --------------------- Error Message >>>>>>>>> -------------------------------------------------------------- >>>>>>>>> [0]PETSC ERROR: Unknown type. Check for miss-spelling or missing >>>>>>>>> package: http://www.mcs.anl.gov/petsc/d >>>>>>>>> ocumentation/installation.html#external >>>>>>>>> [0]PETSC ERROR: Unknown vector type: seqcusp >>>>>>>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/d >>>>>>>>> ocumentation/faq.html for trouble shooting. >>>>>>>>> [0]PETSC ERROR: Petsc Development GIT revision: >>>>>>>>> v3.8.3-2027-g045eeab GIT Date: 2018-03-12 13:30:25 -0500 >>>>>>>>> [0]PETSC ERROR: ./ex19 on a named node50 by valera Wed Mar 14 >>>>>>>>> 11:12:11 2018 >>>>>>>>> [0]PETSC ERROR: Configure options >>>>>>>>> --prefix=/usr/local/petsc.cod/petsc-install >>>>>>>>> --with-mpi-dir=/usr/lib64/openmpi --with-blaslapack-dir=/usr/lib64 >>>>>>>>> COPTFLAGS=-O2 CXXOPTFLAGS=-O2 FOPTFLAGS=-O2 --with-shared-libraries=0 >>>>>>>>> --download-hypre --with-debugging=0 --with-cuda=1 >>>>>>>>> --with-cuda-arch=sm_60 >>>>>>>>> [0]PETSC ERROR: #1 VecSetType() line 42 in >>>>>>>>> /home/valera/testpetsc/src/vec/vec/interface/vecreg.c >>>>>>>>> [0]PETSC ERROR: #2 DMCreateGlobalVector_DA() line 39 in >>>>>>>>> /home/valera/testpetsc/src/dm/impls/da/dadist.c >>>>>>>>> [0]PETSC ERROR: #3 DMCreateGlobalVector() line 865 in >>>>>>>>> /home/valera/testpetsc/src/dm/interface/dm.c >>>>>>>>> [0]PETSC ERROR: #4 main() line 158 in >>>>>>>>> /home/valera/testpetsc/src/snes/examples/tutorials/ex19.c >>>>>>>>> [0]PETSC ERROR: PETSc Option Table entries: >>>>>>>>> [0]PETSC ERROR: -dm_mat_type seqaijcusp >>>>>>>>> [0]PETSC ERROR: -dm_vec_type seqcusp >>>>>>>>> [0]PETSC ERROR: ----------------End of Error Message -------send >>>>>>>>> entire error message to petsc-ma...@mcs.anl.gov---------- >>>>>>>>> ------------------------------------------------------------ >>>>>>>>> -------------- >>>>>>>>> MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD >>>>>>>>> with errorcode 86. >>>>>>>>> >>>>>>>>> NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. >>>>>>>>> You may or may not see output from other processes, depending on >>>>>>>>> exactly when Open MPI kills them. >>>>>>>>> ------------------------------------------------------------ >>>>>>>>> -------------- >>>>>>>>> [valera@node50 tutorials]$ >>>>>>>>> >>>>>>>>> >>>>>>>>> On Wed, Mar 14, 2018 at 11:10 AM, Matthew Knepley < >>>>>>>>> knep...@gmail.com> wrote: >>>>>>>>> >>>>>>>>>> On Thu, Mar 15, 2018 at 2:46 AM, Manuel Valera < >>>>>>>>>> mvaler...@mail.sdsu.edu> wrote: >>>>>>>>>> >>>>>>>>>>> Ok lets try that, if i go to /home/valera/testpetsc/arch >>>>>>>>>>> -linux2-c-opt/tests/src/snes/examples/tutorials there is >>>>>>>>>>> runex19.sh and a lot of other ex19 variantes, but if i run that i >>>>>>>>>>> get: >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> knepley/feature-plex-functionals *$:/PETSc3/petsc/petsc-dev$ >>>>>>>>>> pushd src/snes/examples/tutorials/ >>>>>>>>>> knepley/feature-plex-functionals >>>>>>>>>> *$:/PETSc3/petsc/petsc-dev/src/snes/examples/tutorials$ >>>>>>>>>> PETSC_ARCH=arch-master-debug make ex19 >>>>>>>>>> knepley/feature-plex-functionals >>>>>>>>>> *$:/PETSc3/petsc/petsc-dev/src/snes/examples/tutorials$ >>>>>>>>>> ./ex19 -dm_vec_type seqcusp -dm_mat_type seqaijcusp >>>>>>>>>> >>>>>>>>>> Thanks, >>>>>>>>>> >>>>>>>>>> Matt >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> [valera@node50 tutorials]$ ./runex19.sh >>>>>>>>>>> not ok snes_tutorials-ex19_1 >>>>>>>>>>> # ------------------------------------------------------------ >>>>>>>>>>> -------------- >>>>>>>>>>> # mpiexec was unable to launch the specified application as it >>>>>>>>>>> could not access >>>>>>>>>>> # or execute an executable: >>>>>>>>>>> # >>>>>>>>>>> # Executable: ../ex19 >>>>>>>>>>> # Node: node50 >>>>>>>>>>> # >>>>>>>>>>> # while attempting to start process rank 0. >>>>>>>>>>> # ------------------------------------------------------------ >>>>>>>>>>> -------------- >>>>>>>>>>> # 2 total processes failed to start >>>>>>>>>>> ok snes_tutorials-ex19_1 # SKIP Command failed so no diff >>>>>>>>>>> >>>>>>>>>>> is this the one i should be running ? >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Wed, Mar 14, 2018 at 10:39 AM, Matthew Knepley < >>>>>>>>>>> knep...@gmail.com> wrote: >>>>>>>>>>> >>>>>>>>>>>> On Thu, Mar 15, 2018 at 2:27 AM, Manuel Valera < >>>>>>>>>>>> mvaler...@mail.sdsu.edu> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Ok thanks Matt, i made a smaller case with only the linear >>>>>>>>>>>>> solver and a 25x25 matrix, the error i have in this case is: >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Ah, it appears that not all parts of your problem are taking >>>>>>>>>>>> the type options. If you want the >>>>>>>>>>>> linear algebra objects to change type, you need to have >>>>>>>>>>>> >>>>>>>>>>>> VecSetFromOptions() and MatSetFromOptions() >>>>>>>>>>>> >>>>>>>>>>>> called after you create them, but before sizes are set and data >>>>>>>>>>>> is entered. However, it should >>>>>>>>>>>> not be possible to have a seq Vec with the seqcusp AXPY routine >>>>>>>>>>>> set. Something else is wrong... >>>>>>>>>>>> Did you try a PETSc example, such as SNES ex19, with this? >>>>>>>>>>>> >>>>>>>>>>>> Thanks, >>>>>>>>>>>> >>>>>>>>>>>> Matt >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>> [valera@node50 alone]$ mpirun -n 1 ./linsolve -vec_type cusp >>>>>>>>>>>>> -mat_type aijcusparse >>>>>>>>>>>>> laplacian.petsc ! >>>>>>>>>>>>> TrivSoln loaded, size: 125 / 125 >>>>>>>>>>>>> RHS loaded, size: 125 / 125 >>>>>>>>>>>>> [0]PETSC ERROR: --------------------- Error Message >>>>>>>>>>>>> -------------------------------------------------------------- >>>>>>>>>>>>> [0]PETSC ERROR: Null argument, when expecting valid pointer >>>>>>>>>>>>> [0]PETSC ERROR: Null Pointer: Parameter # 4 >>>>>>>>>>>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/d >>>>>>>>>>>>> ocumentation/faq.html for trouble shooting. >>>>>>>>>>>>> [0]PETSC ERROR: Petsc Development GIT revision: >>>>>>>>>>>>> v3.8.3-1817-g96b6f8a GIT Date: 2018-02-28 10:19:08 -0600 >>>>>>>>>>>>> [0]PETSC ERROR: ./linsolve on a cuda named node50 by valera >>>>>>>>>>>>> Wed Mar 14 10:24:35 2018 >>>>>>>>>>>>> [0]PETSC ERROR: Configure options PETSC_ARCH=cuda >>>>>>>>>>>>> --with-cc=mpicc --with-cxx=mpic++ --with-fc=mpifort >>>>>>>>>>>>> --COPTFLAGS=-O3 >>>>>>>>>>>>> --CXXOPTFLAGS=-O3 --FOPTFLAGS=-O3 --with-shared-libraries=1 >>>>>>>>>>>>> --with-debugging=1 --with-cuda=1 --with-cuda-arch=sm_60 >>>>>>>>>>>>> --with-cusp=1 >>>>>>>>>>>>> --with-cusp-dir=/home/valera/cusp --with-vienacl=1 >>>>>>>>>>>>> --download-fblaslapack=1 --download-hypre >>>>>>>>>>>>> [0]PETSC ERROR: #1 VecSetValues() line 851 in >>>>>>>>>>>>> /home/valera/petsc/src/vec/vec/interface/rvector.c >>>>>>>>>>>>> [0]PETSC ERROR: --------------------- Error Message >>>>>>>>>>>>> -------------------------------------------------------------- >>>>>>>>>>>>> [0]PETSC ERROR: Invalid argument >>>>>>>>>>>>> [0]PETSC ERROR: Object (seq) is not seqcusp or mpicusp >>>>>>>>>>>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/d >>>>>>>>>>>>> ocumentation/faq.html for trouble shooting. >>>>>>>>>>>>> [0]PETSC ERROR: Petsc Development GIT revision: >>>>>>>>>>>>> v3.8.3-1817-g96b6f8a GIT Date: 2018-02-28 10:19:08 -0600 >>>>>>>>>>>>> [0]PETSC ERROR: ./linsolve on a cuda named node50 by valera >>>>>>>>>>>>> Wed Mar 14 10:24:35 2018 >>>>>>>>>>>>> [0]PETSC ERROR: Configure options PETSC_ARCH=cuda >>>>>>>>>>>>> --with-cc=mpicc --with-cxx=mpic++ --with-fc=mpifort >>>>>>>>>>>>> --COPTFLAGS=-O3 >>>>>>>>>>>>> --CXXOPTFLAGS=-O3 --FOPTFLAGS=-O3 --with-shared-libraries=1 >>>>>>>>>>>>> --with-debugging=1 --with-cuda=1 --with-cuda-arch=sm_60 >>>>>>>>>>>>> --with-cusp=1 >>>>>>>>>>>>> --with-cusp-dir=/home/valera/cusp --with-vienacl=1 >>>>>>>>>>>>> --download-fblaslapack=1 --download-hypre >>>>>>>>>>>>> [0]PETSC ERROR: #2 VecCUSPGetArrayRead() line 1792 in >>>>>>>>>>>>> /home/valera/petsc/src/vec/vec/impls/seq/seqcusp/veccusp2.cu >>>>>>>>>>>>> [0]PETSC ERROR: #3 VecAXPY_SeqCUSP() line 314 in >>>>>>>>>>>>> /home/valera/petsc/src/vec/vec/impls/seq/seqcusp/veccusp2.cu >>>>>>>>>>>>> [0]PETSC ERROR: #4 VecAXPY() line 612 in >>>>>>>>>>>>> /home/valera/petsc/src/vec/vec/interface/rvector.c >>>>>>>>>>>>> [0]PETSC ERROR: #5 KSPSolve_GCR_cycle() line 60 in >>>>>>>>>>>>> /home/valera/petsc/src/ksp/ksp/impls/gcr/gcr.c >>>>>>>>>>>>> [0]PETSC ERROR: #6 KSPSolve_GCR() line 114 in >>>>>>>>>>>>> /home/valera/petsc/src/ksp/ksp/impls/gcr/gcr.c >>>>>>>>>>>>> [0]PETSC ERROR: #7 KSPSolve() line 669 in >>>>>>>>>>>>> /home/valera/petsc/src/ksp/ksp/interface/itfunc.c >>>>>>>>>>>>> soln maxval: 0.0000000000000000 >>>>>>>>>>>>> soln minval: 0.0000000000000000 >>>>>>>>>>>>> Norm: 11.180339887498949 >>>>>>>>>>>>> Its: 0 >>>>>>>>>>>>> WARNING! There are options you set that were not used! >>>>>>>>>>>>> WARNING! could be spelling mistake, etc! >>>>>>>>>>>>> Option left: name:-mat_type value: aijcusparse >>>>>>>>>>>>> [valera@node50 alone]$ >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> I also see the configure options are not correct, so i guess >>>>>>>>>>>>> is still linking a different petsc installation, but maybe we can >>>>>>>>>>>>> try to >>>>>>>>>>>>> make it work as it is, i will let you know if i am able to link >>>>>>>>>>>>> the correct >>>>>>>>>>>>> petsc installation here, >>>>>>>>>>>>> >>>>>>>>>>>>> Best, >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> On Sun, Mar 11, 2018 at 9:00 AM, Matthew Knepley < >>>>>>>>>>>>> knep...@gmail.com> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> On Fri, Mar 9, 2018 at 3:05 AM, Manuel Valera < >>>>>>>>>>>>>> mvaler...@mail.sdsu.edu> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> Hello all, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I am working on porting a linear solver into GPUs for timing >>>>>>>>>>>>>>> purposes, so far i've been able to compile and run the CUSP >>>>>>>>>>>>>>> libraries and >>>>>>>>>>>>>>> compile PETSc to be used with CUSP and ViennaCL, after the >>>>>>>>>>>>>>> initial runs i >>>>>>>>>>>>>>> noticed some errors, they are different for different flags and >>>>>>>>>>>>>>> i would >>>>>>>>>>>>>>> appreciate any help interpreting them, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> The only elements in this program that use PETSc are the >>>>>>>>>>>>>>> laplacian matrix (sparse), the RHS and X vectors and a scatter >>>>>>>>>>>>>>> petsc >>>>>>>>>>>>>>> object, so i would say it's safe to pass the command line >>>>>>>>>>>>>>> arguments for the >>>>>>>>>>>>>>> Mat/VecSetType()s instead of changing the source code, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> If i use *-vec_type cuda -mat_type aijcusparse* or *-vec_type >>>>>>>>>>>>>>> viennacl -mat_type aijviennacl *i get the following: >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> These systems do not properly propagate errors. My only >>>>>>>>>>>>>> advice is to run a smaller problem and see. >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>>> [0]PETSC ERROR: ------------------------------ >>>>>>>>>>>>>>> ------------------------------------------ >>>>>>>>>>>>>>> [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation >>>>>>>>>>>>>>> Violation, probably memory access out of range >>>>>>>>>>>>>>> [0]PETSC ERROR: Try option -start_in_debugger or >>>>>>>>>>>>>>> -on_error_attach_debugger >>>>>>>>>>>>>>> [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/d >>>>>>>>>>>>>>> ocumentation/faq.html#valgrind >>>>>>>>>>>>>>> [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and >>>>>>>>>>>>>>> Apple Mac OS X to find memory corruption errors >>>>>>>>>>>>>>> [0]PETSC ERROR: likely location of problem given in stack >>>>>>>>>>>>>>> below >>>>>>>>>>>>>>> [0]PETSC ERROR: --------------------- Stack Frames >>>>>>>>>>>>>>> ------------------------------------ >>>>>>>>>>>>>>> [0]PETSC ERROR: Note: The EXACT line numbers in the stack >>>>>>>>>>>>>>> are not available, >>>>>>>>>>>>>>> [0]PETSC ERROR: INSTEAD the line number of the start >>>>>>>>>>>>>>> of the function >>>>>>>>>>>>>>> [0]PETSC ERROR: is given. >>>>>>>>>>>>>>> [0]PETSC ERROR: [0] VecSetValues line 847 >>>>>>>>>>>>>>> /home/valera/petsc/src/vec/vec/interface/rvector.c >>>>>>>>>>>>>>> [0]PETSC ERROR: [0] VecSetType line 36 >>>>>>>>>>>>>>> /home/valera/petsc/src/vec/vec/interface/vecreg.c >>>>>>>>>>>>>>> [0]PETSC ERROR: [0] VecSetTypeFromOptions_Private line 1230 >>>>>>>>>>>>>>> /home/valera/petsc/src/vec/vec/interface/vector.c >>>>>>>>>>>>>>> [0]PETSC ERROR: [0] VecSetFromOptions line 1271 >>>>>>>>>>>>>>> /home/valera/petsc/src/vec/vec/interface/vector.c >>>>>>>>>>>>>>> [0]PETSC ERROR: --------------------- Error Message >>>>>>>>>>>>>>> ------------------------------------------------------------ >>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>> [0]PETSC ERROR: Signal received >>>>>>>>>>>>>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/d >>>>>>>>>>>>>>> ocumentation/faq.html for trouble shooting. >>>>>>>>>>>>>>> [0]PETSC ERROR: Petsc Development GIT revision: >>>>>>>>>>>>>>> v3.8.3-1817-g96b6f8a GIT Date: 2018-02-28 10:19:08 -0600 >>>>>>>>>>>>>>> [0]PETSC ERROR: ./gcmSeamount on a cuda named node50 by >>>>>>>>>>>>>>> valera Thu Mar 8 09:50:51 2018 >>>>>>>>>>>>>>> [0]PETSC ERROR: Configure options PETSC_ARCH=cuda >>>>>>>>>>>>>>> --with-cc=mpicc --with-cxx=mpic++ --with-fc=mpifort >>>>>>>>>>>>>>> --COPTFLAGS=-O3 >>>>>>>>>>>>>>> --CXXOPTFLAGS=-O3 --FOPTFLAGS=-O3 --with-shared-libraries=1 >>>>>>>>>>>>>>> --with-debugging=1 --with-cuda=1 --with-cuda-arch=sm_60 >>>>>>>>>>>>>>> --with-cusp=1 >>>>>>>>>>>>>>> --with-cusp-dir=/home/valera/cusp --with-vienacl=1 >>>>>>>>>>>>>>> --download-fblaslapack=1 --download-hypre >>>>>>>>>>>>>>> [0]PETSC ERROR: #5 User provided function() line 0 in >>>>>>>>>>>>>>> unknown file >>>>>>>>>>>>>>> ------------------------------------------------------------ >>>>>>>>>>>>>>> -------------- >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> This seems to be a memory out of range, maybe my vector is >>>>>>>>>>>>>>> too big for my CUDA system? how do i assess that? >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Next, if i use *-vec_type cusp -mat_type aijcusparse *i get >>>>>>>>>>>>>>> something different and more interesting: >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> We need to see the entire error message, since it has the >>>>>>>>>>>>>> stack. >>>>>>>>>>>>>> >>>>>>>>>>>>>> This seems like a logic error, but could definitely be on our >>>>>>>>>>>>>> end. Here is how I think about these: >>>>>>>>>>>>>> >>>>>>>>>>>>>> 1) We have nightly test solves, so at least some solver >>>>>>>>>>>>>> configuration works >>>>>>>>>>>>>> >>>>>>>>>>>>>> 2) Some vector which is marked read-only (happens for input >>>>>>>>>>>>>> to solvers), but someone is trying to update it. >>>>>>>>>>>>>> The stack will tell me where this is happening. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>> >>>>>>>>>>>>>> Matt >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>>> [0]PETSC ERROR: --------------------- Error Message >>>>>>>>>>>>>>> ------------------------------------------------------------ >>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>> [0]PETSC ERROR: Object is in wrong state >>>>>>>>>>>>>>> [0]PETSC ERROR: Vec is locked read only, argument # 3 >>>>>>>>>>>>>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/d >>>>>>>>>>>>>>> ocumentation/faq.html for trouble shooting. >>>>>>>>>>>>>>> [0]PETSC ERROR: Petsc Development GIT revision: >>>>>>>>>>>>>>> v3.8.3-1817-g96b6f8a GIT Date: 2018-02-28 10:19:08 -0600 >>>>>>>>>>>>>>> [0]PETSC ERROR: ./gcmSeamount on a cuda named node50 by >>>>>>>>>>>>>>> valera Thu Mar 8 10:02:19 2018 >>>>>>>>>>>>>>> [0]PETSC ERROR: Configure options PETSC_ARCH=cuda >>>>>>>>>>>>>>> --with-cc=mpicc --with-cxx=mpic++ --with-fc=mpifort >>>>>>>>>>>>>>> --COPTFLAGS=-O3 >>>>>>>>>>>>>>> --CXXOPTFLAGS=-O3 --FOPTFLAGS=-O3 --with-shared-libraries=1 >>>>>>>>>>>>>>> --with-debugging=1 --with-cuda=1 --with-cuda-arch=sm_60 >>>>>>>>>>>>>>> --with-cusp=1 >>>>>>>>>>>>>>> --with-cusp-dir=/home/valera/cusp --with-vienacl=1 >>>>>>>>>>>>>>> --download-fblaslapack=1 --download-hypre >>>>>>>>>>>>>>> [0]PETSC ERROR: #48 KSPSolve() line 615 in >>>>>>>>>>>>>>> /home/valera/petsc/src/ksp/ksp/interface/itfunc.c >>>>>>>>>>>>>>> PETSC_SOLVER_ONLY 6.8672990892082453E-005 s >>>>>>>>>>>>>>> [0]PETSC ERROR: --------------------- Error Message >>>>>>>>>>>>>>> ------------------------------------------------------------ >>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>> [0]PETSC ERROR: Invalid argument >>>>>>>>>>>>>>> [0]PETSC ERROR: Object (seq) is not seqcusp or mpicusp >>>>>>>>>>>>>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/d >>>>>>>>>>>>>>> ocumentation/faq.html for trouble shooting. >>>>>>>>>>>>>>> [0]PETSC ERROR: Petsc Development GIT revision: >>>>>>>>>>>>>>> v3.8.3-1817-g96b6f8a GIT Date: 2018-02-28 10:19:08 -0600 >>>>>>>>>>>>>>> [0]PETSC ERROR: ./gcmSeamount on a cuda named node50 by >>>>>>>>>>>>>>> valera Thu Mar 8 10:02:19 2018 >>>>>>>>>>>>>>> [0]PETSC ERROR: Configure options PETSC_ARCH=cuda >>>>>>>>>>>>>>> --with-cc=mpicc --with-cxx=mpic++ --with-fc=mpifort >>>>>>>>>>>>>>> --COPTFLAGS=-O3 >>>>>>>>>>>>>>> --CXXOPTFLAGS=-O3 --FOPTFLAGS=-O3 --with-shared-libraries=1 >>>>>>>>>>>>>>> --with-debugging=1 --with-cuda=1 --with-cuda-arch=sm_60 >>>>>>>>>>>>>>> --with-cusp=1 >>>>>>>>>>>>>>> --with-cusp-dir=/home/valera/cusp --with-vienacl=1 >>>>>>>>>>>>>>> --download-fblaslapack=1 --download-hypre >>>>>>>>>>>>>>> [0]PETSC ERROR: #49 VecCUSPGetArrayReadWrite() line 1718 in >>>>>>>>>>>>>>> /home/valera/petsc/src/vec/vec/impls/seq/seqcusp/veccusp2.cu >>>>>>>>>>>>>>> [0]PETSC ERROR: #50 VecScatterCUSP_StoS() line 269 in >>>>>>>>>>>>>>> /home/valera/petsc/src/vec/vec/impls/seq/seqcusp/vecscatterc >>>>>>>>>>>>>>> usp.cu >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> And it yields a "solution" to the system and also a log at >>>>>>>>>>>>>>> the end: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> ./gcmSeamount on a cuda named node50 with 1 processor, by >>>>>>>>>>>>>>> valera Thu Mar 8 10:02:24 2018 >>>>>>>>>>>>>>> Using Petsc Development GIT revision: v3.8.3-1817-g96b6f8a >>>>>>>>>>>>>>> GIT Date: 2018-02-28 10:19:08 -0600 >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Max Max/Min Avg >>>>>>>>>>>>>>> Total >>>>>>>>>>>>>>> Time (sec): 4.573e+00 1.00000 4.573e+00 >>>>>>>>>>>>>>> Objects: 8.100e+01 1.00000 8.100e+01 >>>>>>>>>>>>>>> Flop: 3.492e+07 1.00000 3.492e+07 >>>>>>>>>>>>>>> 3.492e+07 >>>>>>>>>>>>>>> Flop/sec: 7.637e+06 1.00000 7.637e+06 >>>>>>>>>>>>>>> 7.637e+06 >>>>>>>>>>>>>>> Memory: 2.157e+08 1.00000 >>>>>>>>>>>>>>> 2.157e+08 >>>>>>>>>>>>>>> MPI Messages: 0.000e+00 0.00000 0.000e+00 >>>>>>>>>>>>>>> 0.000e+00 >>>>>>>>>>>>>>> MPI Message Lengths: 0.000e+00 0.00000 0.000e+00 >>>>>>>>>>>>>>> 0.000e+00 >>>>>>>>>>>>>>> MPI Reductions: 0.000e+00 0.00000 >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Flop counting convention: 1 flop = 1 real number operation >>>>>>>>>>>>>>> of type (multiply/divide/add/subtract) >>>>>>>>>>>>>>> e.g., VecAXPY() for real vectors >>>>>>>>>>>>>>> of length N --> 2N flop >>>>>>>>>>>>>>> and VecAXPY() for complex >>>>>>>>>>>>>>> vectors of length N --> 8N flop >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Summary of Stages: ----- Time ------ ----- Flop ----- >>>>>>>>>>>>>>> --- Messages --- -- Message Lengths -- -- Reductions -- >>>>>>>>>>>>>>> Avg %Total Avg %Total >>>>>>>>>>>>>>> counts %Total Avg %Total counts %Total >>>>>>>>>>>>>>> 0: Main Stage: 4.5729e+00 100.0% 3.4924e+07 100.0% >>>>>>>>>>>>>>> 0.000e+00 0.0% 0.000e+00 0.0% 0.000e+00 0.0% >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> ------------------------------------------------------------ >>>>>>>>>>>>>>> ------------------------------------------------------------ >>>>>>>>>>>>>>> See the 'Profiling' chapter of the users' manual for details >>>>>>>>>>>>>>> on interpreting output. >>>>>>>>>>>>>>> Phase summary info: >>>>>>>>>>>>>>> Count: number of times phase was executed >>>>>>>>>>>>>>> Time and Flop: Max - maximum over all processors >>>>>>>>>>>>>>> Ratio - ratio of maximum to minimum over >>>>>>>>>>>>>>> all processors >>>>>>>>>>>>>>> Mess: number of messages sent >>>>>>>>>>>>>>> Avg. len: average message length (bytes) >>>>>>>>>>>>>>> Reduct: number of global reductions >>>>>>>>>>>>>>> Global: entire computation >>>>>>>>>>>>>>> Stage: stages of a computation. Set stages with >>>>>>>>>>>>>>> PetscLogStagePush() and PetscLogStagePop(). >>>>>>>>>>>>>>> %T - percent time in this phase %F - percent >>>>>>>>>>>>>>> flop in this phase >>>>>>>>>>>>>>> %M - percent messages in this phase %L - percent >>>>>>>>>>>>>>> message lengths in this phase >>>>>>>>>>>>>>> %R - percent reductions in this phase >>>>>>>>>>>>>>> Total Mflop/s: 10e-6 * (sum of flop over all >>>>>>>>>>>>>>> processors)/(max time over all processors) >>>>>>>>>>>>>>> ------------------------------------------------------------ >>>>>>>>>>>>>>> ------------------------------------------------------------ >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> ############################## >>>>>>>>>>>>>>> ############################ >>>>>>>>>>>>>>> # >>>>>>>>>>>>>>> # >>>>>>>>>>>>>>> # WARNING!!! >>>>>>>>>>>>>>> # >>>>>>>>>>>>>>> # >>>>>>>>>>>>>>> # >>>>>>>>>>>>>>> # This code was compiled with a debugging option, >>>>>>>>>>>>>>> # >>>>>>>>>>>>>>> # To get timing results run ./configure >>>>>>>>>>>>>>> # >>>>>>>>>>>>>>> # using --with-debugging=no, the performance will >>>>>>>>>>>>>>> # >>>>>>>>>>>>>>> # be generally two or three times faster. >>>>>>>>>>>>>>> # >>>>>>>>>>>>>>> # >>>>>>>>>>>>>>> # >>>>>>>>>>>>>>> ############################## >>>>>>>>>>>>>>> ############################ >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Event Count Time (sec) Flop >>>>>>>>>>>>>>> --- Global --- --- Stage --- Total >>>>>>>>>>>>>>> Max Ratio Max Ratio Max Ratio >>>>>>>>>>>>>>> Mess Avg len Reduct %T %F %M %L %R %T %F %M %L %R Mflop/s >>>>>>>>>>>>>>> ------------------------------------------------------------ >>>>>>>>>>>>>>> ------------------------------------------------------------ >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> --- Event Stage 0: Main Stage >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> MatLUFactorNum 1 1.0 4.9502e-02 1.0 3.49e+07 1.0 >>>>>>>>>>>>>>> 0.0e+00 0.0e+00 0.0e+00 1100 0 0 0 1100 0 0 0 706 >>>>>>>>>>>>>>> MatILUFactorSym 1 1.0 1.9642e-02 1.0 0.00e+00 0.0 >>>>>>>>>>>>>>> 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>>>>>>>>>>>>>> MatAssemblyBegin 2 1.0 6.9141e-06 1.0 0.00e+00 0.0 >>>>>>>>>>>>>>> 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>>>>>>>>>>>>>> MatAssemblyEnd 2 1.0 2.6612e-01 1.0 0.00e+00 0.0 >>>>>>>>>>>>>>> 0.0e+00 0.0e+00 0.0e+00 6 0 0 0 0 6 0 0 0 0 0 >>>>>>>>>>>>>>> MatGetRowIJ 1 1.0 5.0068e-06 1.0 0.00e+00 0.0 >>>>>>>>>>>>>>> 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>>>>>>>>>>>>>> MatGetOrdering 1 1.0 1.7186e-02 1.0 0.00e+00 0.0 >>>>>>>>>>>>>>> 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>>>>>>>>>>>>>> MatLoad 1 1.0 1.1575e-01 1.0 0.00e+00 0.0 >>>>>>>>>>>>>>> 0.0e+00 0.0e+00 0.0e+00 3 0 0 0 0 3 0 0 0 0 0 >>>>>>>>>>>>>>> MatView 1 1.0 8.0877e-02 1.0 0.00e+00 0.0 >>>>>>>>>>>>>>> 0.0e+00 0.0e+00 0.0e+00 2 0 0 0 0 2 0 0 0 0 0 >>>>>>>>>>>>>>> MatCUSPCopyTo 1 1.0 2.4664e-01 1.0 0.00e+00 0.0 >>>>>>>>>>>>>>> 0.0e+00 0.0e+00 0.0e+00 5 0 0 0 0 5 0 0 0 0 0 >>>>>>>>>>>>>>> VecSet 68 1.0 5.1665e-02 1.0 0.00e+00 0.0 >>>>>>>>>>>>>>> 0.0e+00 0.0e+00 0.0e+00 1 0 0 0 0 1 0 0 0 0 0 >>>>>>>>>>>>>>> VecAssemblyBegin 17 1.0 5.2691e-05 1.0 0.00e+00 0.0 >>>>>>>>>>>>>>> 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>>>>>>>>>>>>>> VecAssemblyEnd 17 1.0 4.3631e-05 1.0 0.00e+00 0.0 >>>>>>>>>>>>>>> 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>>>>>>>>>>>>>> VecScatterBegin 15 1.0 1.5345e-03 1.0 0.00e+00 0.0 >>>>>>>>>>>>>>> 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>>>>>>>>>>>>>> VecCUSPCopyFrom 1 1.0 1.1199e-03 1.0 0.00e+00 0.0 >>>>>>>>>>>>>>> 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>>>>>>>>>>>>>> KSPSetUp 1 1.0 5.1929e-02 1.0 0.00e+00 0.0 >>>>>>>>>>>>>>> 0.0e+00 0.0e+00 0.0e+00 1 0 0 0 0 1 0 0 0 0 0 >>>>>>>>>>>>>>> PCSetUp 2 1.0 8.6590e-02 1.0 3.49e+07 1.0 >>>>>>>>>>>>>>> 0.0e+00 0.0e+00 0.0e+00 2100 0 0 0 2100 0 0 0 403 >>>>>>>>>>>>>>> ------------------------------------------------------------ >>>>>>>>>>>>>>> ------------------------------------------------------------ >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Memory usage is given in bytes: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Object Type Creations Destructions Memory >>>>>>>>>>>>>>> Descendants' Mem. >>>>>>>>>>>>>>> Reports information only for process 0. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> --- Event Stage 0: Main Stage >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Matrix 3 1 52856972 0. >>>>>>>>>>>>>>> Matrix Null Space 1 1 608 0. >>>>>>>>>>>>>>> Vector 66 3 3414600 0. >>>>>>>>>>>>>>> Vector Scatter 1 1 680 0. >>>>>>>>>>>>>>> Viewer 3 2 1680 0. >>>>>>>>>>>>>>> Krylov Solver 1 0 0 0. >>>>>>>>>>>>>>> Preconditioner 2 1 864 0. >>>>>>>>>>>>>>> Index Set 4 1 800 0. >>>>>>>>>>>>>>> ============================================================ >>>>>>>>>>>>>>> ============================================================ >>>>>>>>>>>>>>> Average time to get PetscTime(): 9.53674e-08 >>>>>>>>>>>>>>> #PETSc Option Table entries: >>>>>>>>>>>>>>> -ksp_view >>>>>>>>>>>>>>> -log_view >>>>>>>>>>>>>>> -mat_type aijcusparse >>>>>>>>>>>>>>> -matload_block_size 1 >>>>>>>>>>>>>>> -vec_type cusp >>>>>>>>>>>>>>> #End of PETSc Option Table entries >>>>>>>>>>>>>>> Compiled without FORTRAN kernels >>>>>>>>>>>>>>> Compiled with full precision matrices (default) >>>>>>>>>>>>>>> sizeof(short) 2 sizeof(int) 4 sizeof(long) 8 sizeof(void*) 8 >>>>>>>>>>>>>>> sizeof(PetscScalar) 8 sizeof(PetscInt) 4 >>>>>>>>>>>>>>> Configure options: PETSC_ARCH=cuda --with-cc=mpicc >>>>>>>>>>>>>>> --with-cxx=mpic++ --with-fc=mpifort --COPTFLAGS=-O3 >>>>>>>>>>>>>>> --CXXOPTFLAGS=-O3 >>>>>>>>>>>>>>> --FOPTFLAGS=-O3 --with-shared-libraries=1 --with-debugging=1 >>>>>>>>>>>>>>> --with-cuda=1 >>>>>>>>>>>>>>> --with-cuda-arch=sm_60 --with-cusp=1 >>>>>>>>>>>>>>> --with-cusp-dir=/home/valera/cusp >>>>>>>>>>>>>>> --with-vienacl=1 --download-fblaslapack=1 --download-hypre >>>>>>>>>>>>>>> ----------------------------------------- >>>>>>>>>>>>>>> Libraries compiled on Mon Mar 5 16:37:18 2018 on node50 >>>>>>>>>>>>>>> Machine characteristics: Linux-3.10.0-693.17.1.el7.x86_ >>>>>>>>>>>>>>> 64-x86_64-with-centos-7.2.1511-Core >>>>>>>>>>>>>>> Using PETSc directory: /home/valera/petsc >>>>>>>>>>>>>>> Using PETSc arch: cuda >>>>>>>>>>>>>>> ----------------------------------------- >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Using C compiler: mpicc -fPIC -Wall -Wwrite-strings >>>>>>>>>>>>>>> -Wno-strict-aliasing -Wno-unknown-pragmas -fstack-protector >>>>>>>>>>>>>>> -fvisibility=hidden -O3 >>>>>>>>>>>>>>> Using Fortran compiler: mpifort -fPIC -Wall >>>>>>>>>>>>>>> -ffree-line-length-0 -Wno-unused-dummy-argument -O3 >>>>>>>>>>>>>>> ----------------------------------------- >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Using include paths: -I/home/valera/petsc/cuda/include >>>>>>>>>>>>>>> -I/home/valera/petsc/include -I/home/valera/petsc/include >>>>>>>>>>>>>>> -I/home/valera/petsc/cuda/include -I/home/valera/cusp/ >>>>>>>>>>>>>>> -I/usr/local/cuda/include >>>>>>>>>>>>>>> ----------------------------------------- >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Using C linker: mpicc >>>>>>>>>>>>>>> Using Fortran linker: mpifort >>>>>>>>>>>>>>> Using libraries: -Wl,-rpath,/home/valera/petsc/cuda/lib >>>>>>>>>>>>>>> -L/home/valera/petsc/cuda/lib -lpetsc >>>>>>>>>>>>>>> -Wl,-rpath,/home/valera/petsc/cuda/lib >>>>>>>>>>>>>>> -L/home/valera/petsc/cuda/lib -Wl,-rpath,/usr/local/cuda/lib64 >>>>>>>>>>>>>>> -L/usr/local/cuda/lib64 -Wl,-rpath,/usr/lib64/openmpi/lib >>>>>>>>>>>>>>> -L/usr/lib64/openmpi/lib >>>>>>>>>>>>>>> -Wl,-rpath,/usr/lib/gcc/x86_64-redhat-linux/4.8.5 >>>>>>>>>>>>>>> -L/usr/lib/gcc/x86_64-redhat-linux/4.8.5 -lHYPRE -lflapack >>>>>>>>>>>>>>> -lfblas -lm -lcufft -lcublas -lcudart -lcusparse -lX11 -lstdc++ >>>>>>>>>>>>>>> -ldl >>>>>>>>>>>>>>> -lmpi_usempi -lmpi_mpifh -lmpi -lgfortran -lm -lgfortran -lm >>>>>>>>>>>>>>> -lgcc_s >>>>>>>>>>>>>>> -lquadmath -lpthread -lstdc++ -ldl >>>>>>>>>>>>>>> ----------------------------------------- >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Thanks for your help, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Manuel >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> -- >>>>>>>>>>>>>> What most experimenters take for granted before they begin >>>>>>>>>>>>>> their experiments is infinitely more interesting than any >>>>>>>>>>>>>> results to which >>>>>>>>>>>>>> their experiments lead. >>>>>>>>>>>>>> -- Norbert Wiener >>>>>>>>>>>>>> >>>>>>>>>>>>>> https://www.cse.buffalo.edu/~knepley/ >>>>>>>>>>>>>> <http://www.caam.rice.edu/~mk51/> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>>>> What most experimenters take for granted before they begin >>>>>>>>>>>> their experiments is infinitely more interesting than any results >>>>>>>>>>>> to which >>>>>>>>>>>> their experiments lead. >>>>>>>>>>>> -- Norbert Wiener >>>>>>>>>>>> >>>>>>>>>>>> https://www.cse.buffalo.edu/~knepley/ >>>>>>>>>>>> <http://www.caam.rice.edu/~mk51/> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> What most experimenters take for granted before they begin their >>>>>>>>>> experiments is infinitely more interesting than any results to which >>>>>>>>>> their >>>>>>>>>> experiments lead. >>>>>>>>>> -- Norbert Wiener >>>>>>>>>> >>>>>>>>>> https://www.cse.buffalo.edu/~knepley/ >>>>>>>>>> <http://www.caam.rice.edu/~mk51/> >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> What most experimenters take for granted before they begin their >>>>>>>> experiments is infinitely more interesting than any results to which >>>>>>>> their >>>>>>>> experiments lead. >>>>>>>> -- Norbert Wiener >>>>>>>> >>>>>>>> https://www.cse.buffalo.edu/~knepley/ >>>>>>>> <http://www.caam.rice.edu/~mk51/> >>>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> What most experimenters take for granted before they begin their >>>>>> experiments is infinitely more interesting than any results to which >>>>>> their >>>>>> experiments lead. >>>>>> -- Norbert Wiener >>>>>> >>>>>> https://www.cse.buffalo.edu/~knepley/ >>>>>> <http://www.caam.rice.edu/~mk51/> >>>>>> >>>>> >>>>> >>>> >>>> >>>> -- >>>> What most experimenters take for granted before they begin their >>>> experiments is infinitely more interesting than any results to which their >>>> experiments lead. >>>> -- Norbert Wiener >>>> >>>> https://www.cse.buffalo.edu/~knepley/ <http://www.caam.rice.edu/~mk51/> >>>> >>> >>> >> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> >> https://www.cse.buffalo.edu/~knepley/ <http://www.caam.rice.edu/~mk51/> >> > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ <http://www.caam.rice.edu/~mk51/>