Ok lets try that, if i go to /home/valera/testpetsc/arch-linux2-c-opt/tests/src/snes/examples/tutorials there is runex19.sh and a lot of other ex19 variantes, but if i run that i get:
[valera@node50 tutorials]$ ./runex19.sh not ok snes_tutorials-ex19_1 # -------------------------------------------------------------------------- # mpiexec was unable to launch the specified application as it could not access # or execute an executable: # # Executable: ../ex19 # Node: node50 # # while attempting to start process rank 0. # -------------------------------------------------------------------------- # 2 total processes failed to start ok snes_tutorials-ex19_1 # SKIP Command failed so no diff is this the one i should be running ? On Wed, Mar 14, 2018 at 10:39 AM, Matthew Knepley <knep...@gmail.com> wrote: > On Thu, Mar 15, 2018 at 2:27 AM, Manuel Valera <mvaler...@mail.sdsu.edu> > wrote: > >> Ok thanks Matt, i made a smaller case with only the linear solver and a >> 25x25 matrix, the error i have in this case is: >> > > Ah, it appears that not all parts of your problem are taking the type > options. If you want the > linear algebra objects to change type, you need to have > > VecSetFromOptions() and MatSetFromOptions() > > called after you create them, but before sizes are set and data is > entered. However, it should > not be possible to have a seq Vec with the seqcusp AXPY routine set. > Something else is wrong... > Did you try a PETSc example, such as SNES ex19, with this? > > Thanks, > > Matt > > >> [valera@node50 alone]$ mpirun -n 1 ./linsolve -vec_type cusp -mat_type >> aijcusparse >> laplacian.petsc ! >> TrivSoln loaded, size: 125 / 125 >> RHS loaded, size: 125 / 125 >> [0]PETSC ERROR: --------------------- Error Message >> -------------------------------------------------------------- >> [0]PETSC ERROR: Null argument, when expecting valid pointer >> [0]PETSC ERROR: Null Pointer: Parameter # 4 >> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html >> for trouble shooting. >> [0]PETSC ERROR: Petsc Development GIT revision: v3.8.3-1817-g96b6f8a GIT >> Date: 2018-02-28 10:19:08 -0600 >> [0]PETSC ERROR: ./linsolve on a cuda named node50 by valera Wed Mar 14 >> 10:24:35 2018 >> [0]PETSC ERROR: Configure options PETSC_ARCH=cuda --with-cc=mpicc >> --with-cxx=mpic++ --with-fc=mpifort --COPTFLAGS=-O3 --CXXOPTFLAGS=-O3 >> --FOPTFLAGS=-O3 --with-shared-libraries=1 --with-debugging=1 --with-cuda=1 >> --with-cuda-arch=sm_60 --with-cusp=1 --with-cusp-dir=/home/valera/cusp >> --with-vienacl=1 --download-fblaslapack=1 --download-hypre >> [0]PETSC ERROR: #1 VecSetValues() line 851 in >> /home/valera/petsc/src/vec/vec/interface/rvector.c >> [0]PETSC ERROR: --------------------- Error Message >> -------------------------------------------------------------- >> [0]PETSC ERROR: Invalid argument >> [0]PETSC ERROR: Object (seq) is not seqcusp or mpicusp >> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html >> for trouble shooting. >> [0]PETSC ERROR: Petsc Development GIT revision: v3.8.3-1817-g96b6f8a GIT >> Date: 2018-02-28 10:19:08 -0600 >> [0]PETSC ERROR: ./linsolve on a cuda named node50 by valera Wed Mar 14 >> 10:24:35 2018 >> [0]PETSC ERROR: Configure options PETSC_ARCH=cuda --with-cc=mpicc >> --with-cxx=mpic++ --with-fc=mpifort --COPTFLAGS=-O3 --CXXOPTFLAGS=-O3 >> --FOPTFLAGS=-O3 --with-shared-libraries=1 --with-debugging=1 --with-cuda=1 >> --with-cuda-arch=sm_60 --with-cusp=1 --with-cusp-dir=/home/valera/cusp >> --with-vienacl=1 --download-fblaslapack=1 --download-hypre >> [0]PETSC ERROR: #2 VecCUSPGetArrayRead() line 1792 in >> /home/valera/petsc/src/vec/vec/impls/seq/seqcusp/veccusp2.cu >> [0]PETSC ERROR: #3 VecAXPY_SeqCUSP() line 314 in >> /home/valera/petsc/src/vec/vec/impls/seq/seqcusp/veccusp2.cu >> [0]PETSC ERROR: #4 VecAXPY() line 612 in /home/valera/petsc/src/vec/vec >> /interface/rvector.c >> [0]PETSC ERROR: #5 KSPSolve_GCR_cycle() line 60 in >> /home/valera/petsc/src/ksp/ksp/impls/gcr/gcr.c >> [0]PETSC ERROR: #6 KSPSolve_GCR() line 114 in >> /home/valera/petsc/src/ksp/ksp/impls/gcr/gcr.c >> [0]PETSC ERROR: #7 KSPSolve() line 669 in /home/valera/petsc/src/ksp/ksp >> /interface/itfunc.c >> soln maxval: 0.0000000000000000 >> soln minval: 0.0000000000000000 >> Norm: 11.180339887498949 >> Its: 0 >> WARNING! There are options you set that were not used! >> WARNING! could be spelling mistake, etc! >> Option left: name:-mat_type value: aijcusparse >> [valera@node50 alone]$ >> >> >> I also see the configure options are not correct, so i guess is still >> linking a different petsc installation, but maybe we can try to make it >> work as it is, i will let you know if i am able to link the correct petsc >> installation here, >> >> Best, >> >> >> >> >> On Sun, Mar 11, 2018 at 9:00 AM, Matthew Knepley <knep...@gmail.com> >> wrote: >> >>> On Fri, Mar 9, 2018 at 3:05 AM, Manuel Valera <mvaler...@mail.sdsu.edu> >>> wrote: >>> >>>> Hello all, >>>> >>>> I am working on porting a linear solver into GPUs for timing purposes, >>>> so far i've been able to compile and run the CUSP libraries and compile >>>> PETSc to be used with CUSP and ViennaCL, after the initial runs i noticed >>>> some errors, they are different for different flags and i would appreciate >>>> any help interpreting them, >>>> >>>> The only elements in this program that use PETSc are the laplacian >>>> matrix (sparse), the RHS and X vectors and a scatter petsc object, so i >>>> would say it's safe to pass the command line arguments for the >>>> Mat/VecSetType()s instead of changing the source code, >>>> >>>> If i use *-vec_type cuda -mat_type aijcusparse* or *-vec_type viennacl >>>> -mat_type aijviennacl *i get the following: >>>> >>> >>> These systems do not properly propagate errors. My only advice is to run >>> a smaller problem and see. >>> >>> >>>> [0]PETSC ERROR: ------------------------------ >>>> ------------------------------------------ >>>> [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, >>>> probably memory access out of range >>>> [0]PETSC ERROR: Try option -start_in_debugger or >>>> -on_error_attach_debugger >>>> [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/d >>>> ocumentation/faq.html#valgrind >>>> [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac >>>> OS X to find memory corruption errors >>>> [0]PETSC ERROR: likely location of problem given in stack below >>>> [0]PETSC ERROR: --------------------- Stack Frames >>>> ------------------------------------ >>>> [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not >>>> available, >>>> [0]PETSC ERROR: INSTEAD the line number of the start of the >>>> function >>>> [0]PETSC ERROR: is given. >>>> [0]PETSC ERROR: [0] VecSetValues line 847 /home/valera/petsc/src/vec/vec >>>> /interface/rvector.c >>>> [0]PETSC ERROR: [0] VecSetType line 36 /home/valera/petsc/src/vec/vec >>>> /interface/vecreg.c >>>> [0]PETSC ERROR: [0] VecSetTypeFromOptions_Private line 1230 >>>> /home/valera/petsc/src/vec/vec/interface/vector.c >>>> [0]PETSC ERROR: [0] VecSetFromOptions line 1271 >>>> /home/valera/petsc/src/vec/vec/interface/vector.c >>>> [0]PETSC ERROR: --------------------- Error Message >>>> -------------------------------------------------------------- >>>> [0]PETSC ERROR: Signal received >>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html >>>> for trouble shooting. >>>> [0]PETSC ERROR: Petsc Development GIT revision: v3.8.3-1817-g96b6f8a >>>> GIT Date: 2018-02-28 10:19:08 -0600 >>>> [0]PETSC ERROR: ./gcmSeamount on a cuda named node50 by valera Thu Mar >>>> 8 09:50:51 2018 >>>> [0]PETSC ERROR: Configure options PETSC_ARCH=cuda --with-cc=mpicc >>>> --with-cxx=mpic++ --with-fc=mpifort --COPTFLAGS=-O3 --CXXOPTFLAGS=-O3 >>>> --FOPTFLAGS=-O3 --with-shared-libraries=1 --with-debugging=1 --with-cuda=1 >>>> --with-cuda-arch=sm_60 --with-cusp=1 --with-cusp-dir=/home/valera/cusp >>>> --with-vienacl=1 --download-fblaslapack=1 --download-hypre >>>> [0]PETSC ERROR: #5 User provided function() line 0 in unknown file >>>> ------------------------------------------------------------ >>>> -------------- >>>> >>>> This seems to be a memory out of range, maybe my vector is too big for >>>> my CUDA system? how do i assess that? >>>> >>>> >>>> Next, if i use *-vec_type cusp -mat_type aijcusparse *i get something >>>> different and more interesting: >>>> >>> >>> We need to see the entire error message, since it has the stack. >>> >>> This seems like a logic error, but could definitely be on our end. Here >>> is how I think about these: >>> >>> 1) We have nightly test solves, so at least some solver configuration >>> works >>> >>> 2) Some vector which is marked read-only (happens for input to >>> solvers), but someone is trying to update it. >>> The stack will tell me where this is happening. >>> >>> Thanks, >>> >>> Matt >>> >>> >>>> [0]PETSC ERROR: --------------------- Error Message >>>> -------------------------------------------------------------- >>>> [0]PETSC ERROR: Object is in wrong state >>>> [0]PETSC ERROR: Vec is locked read only, argument # 3 >>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html >>>> for trouble shooting. >>>> [0]PETSC ERROR: Petsc Development GIT revision: v3.8.3-1817-g96b6f8a >>>> GIT Date: 2018-02-28 10:19:08 -0600 >>>> [0]PETSC ERROR: ./gcmSeamount on a cuda named node50 by valera Thu Mar >>>> 8 10:02:19 2018 >>>> [0]PETSC ERROR: Configure options PETSC_ARCH=cuda --with-cc=mpicc >>>> --with-cxx=mpic++ --with-fc=mpifort --COPTFLAGS=-O3 --CXXOPTFLAGS=-O3 >>>> --FOPTFLAGS=-O3 --with-shared-libraries=1 --with-debugging=1 --with-cuda=1 >>>> --with-cuda-arch=sm_60 --with-cusp=1 --with-cusp-dir=/home/valera/cusp >>>> --with-vienacl=1 --download-fblaslapack=1 --download-hypre >>>> [0]PETSC ERROR: #48 KSPSolve() line 615 in >>>> /home/valera/petsc/src/ksp/ksp/interface/itfunc.c >>>> PETSC_SOLVER_ONLY 6.8672990892082453E-005 s >>>> [0]PETSC ERROR: --------------------- Error Message >>>> -------------------------------------------------------------- >>>> [0]PETSC ERROR: Invalid argument >>>> [0]PETSC ERROR: Object (seq) is not seqcusp or mpicusp >>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html >>>> for trouble shooting. >>>> [0]PETSC ERROR: Petsc Development GIT revision: v3.8.3-1817-g96b6f8a >>>> GIT Date: 2018-02-28 10:19:08 -0600 >>>> [0]PETSC ERROR: ./gcmSeamount on a cuda named node50 by valera Thu Mar >>>> 8 10:02:19 2018 >>>> [0]PETSC ERROR: Configure options PETSC_ARCH=cuda --with-cc=mpicc >>>> --with-cxx=mpic++ --with-fc=mpifort --COPTFLAGS=-O3 --CXXOPTFLAGS=-O3 >>>> --FOPTFLAGS=-O3 --with-shared-libraries=1 --with-debugging=1 --with-cuda=1 >>>> --with-cuda-arch=sm_60 --with-cusp=1 --with-cusp-dir=/home/valera/cusp >>>> --with-vienacl=1 --download-fblaslapack=1 --download-hypre >>>> [0]PETSC ERROR: #49 VecCUSPGetArrayReadWrite() line 1718 in >>>> /home/valera/petsc/src/vec/vec/impls/seq/seqcusp/veccusp2.cu >>>> [0]PETSC ERROR: #50 VecScatterCUSP_StoS() line 269 in >>>> /home/valera/petsc/src/vec/vec/impls/seq/seqcusp/vecscattercusp.cu >>>> >>>> >>>> >>>> >>>> >>>> And it yields a "solution" to the system and also a log at the end: >>>> >>>> >>>> >>>> >>>> >>>> ./gcmSeamount on a cuda named node50 with 1 processor, by valera Thu >>>> Mar 8 10:02:24 2018 >>>> Using Petsc Development GIT revision: v3.8.3-1817-g96b6f8a GIT Date: >>>> 2018-02-28 10:19:08 -0600 >>>> >>>> Max Max/Min Avg Total >>>> Time (sec): 4.573e+00 1.00000 4.573e+00 >>>> Objects: 8.100e+01 1.00000 8.100e+01 >>>> Flop: 3.492e+07 1.00000 3.492e+07 3.492e+07 >>>> Flop/sec: 7.637e+06 1.00000 7.637e+06 7.637e+06 >>>> Memory: 2.157e+08 1.00000 2.157e+08 >>>> MPI Messages: 0.000e+00 0.00000 0.000e+00 0.000e+00 >>>> MPI Message Lengths: 0.000e+00 0.00000 0.000e+00 0.000e+00 >>>> MPI Reductions: 0.000e+00 0.00000 >>>> >>>> Flop counting convention: 1 flop = 1 real number operation of type >>>> (multiply/divide/add/subtract) >>>> e.g., VecAXPY() for real vectors of length >>>> N --> 2N flop >>>> and VecAXPY() for complex vectors of length >>>> N --> 8N flop >>>> >>>> Summary of Stages: ----- Time ------ ----- Flop ----- --- Messages >>>> --- -- Message Lengths -- -- Reductions -- >>>> Avg %Total Avg %Total counts >>>> %Total Avg %Total counts %Total >>>> 0: Main Stage: 4.5729e+00 100.0% 3.4924e+07 100.0% 0.000e+00 >>>> 0.0% 0.000e+00 0.0% 0.000e+00 0.0% >>>> >>>> ------------------------------------------------------------ >>>> ------------------------------------------------------------ >>>> See the 'Profiling' chapter of the users' manual for details on >>>> interpreting output. >>>> Phase summary info: >>>> Count: number of times phase was executed >>>> Time and Flop: Max - maximum over all processors >>>> Ratio - ratio of maximum to minimum over all >>>> processors >>>> Mess: number of messages sent >>>> Avg. len: average message length (bytes) >>>> Reduct: number of global reductions >>>> Global: entire computation >>>> Stage: stages of a computation. Set stages with PetscLogStagePush() >>>> and PetscLogStagePop(). >>>> %T - percent time in this phase %F - percent flop in this >>>> phase >>>> %M - percent messages in this phase %L - percent message >>>> lengths in this phase >>>> %R - percent reductions in this phase >>>> Total Mflop/s: 10e-6 * (sum of flop over all processors)/(max time >>>> over all processors) >>>> ------------------------------------------------------------ >>>> ------------------------------------------------------------ >>>> >>>> >>>> ########################################################## >>>> # # >>>> # WARNING!!! # >>>> # # >>>> # This code was compiled with a debugging option, # >>>> # To get timing results run ./configure # >>>> # using --with-debugging=no, the performance will # >>>> # be generally two or three times faster. # >>>> # # >>>> ########################################################## >>>> >>>> >>>> Event Count Time (sec) Flop >>>> --- Global --- --- Stage --- Total >>>> Max Ratio Max Ratio Max Ratio Mess Avg >>>> len Reduct %T %F %M %L %R %T %F %M %L %R Mflop/s >>>> ------------------------------------------------------------ >>>> ------------------------------------------------------------ >>>> >>>> --- Event Stage 0: Main Stage >>>> >>>> MatLUFactorNum 1 1.0 4.9502e-02 1.0 3.49e+07 1.0 0.0e+00 >>>> 0.0e+00 0.0e+00 1100 0 0 0 1100 0 0 0 706 >>>> MatILUFactorSym 1 1.0 1.9642e-02 1.0 0.00e+00 0.0 0.0e+00 >>>> 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>>> MatAssemblyBegin 2 1.0 6.9141e-06 1.0 0.00e+00 0.0 0.0e+00 >>>> 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>>> MatAssemblyEnd 2 1.0 2.6612e-01 1.0 0.00e+00 0.0 0.0e+00 >>>> 0.0e+00 0.0e+00 6 0 0 0 0 6 0 0 0 0 0 >>>> MatGetRowIJ 1 1.0 5.0068e-06 1.0 0.00e+00 0.0 0.0e+00 >>>> 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>>> MatGetOrdering 1 1.0 1.7186e-02 1.0 0.00e+00 0.0 0.0e+00 >>>> 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>>> MatLoad 1 1.0 1.1575e-01 1.0 0.00e+00 0.0 0.0e+00 >>>> 0.0e+00 0.0e+00 3 0 0 0 0 3 0 0 0 0 0 >>>> MatView 1 1.0 8.0877e-02 1.0 0.00e+00 0.0 0.0e+00 >>>> 0.0e+00 0.0e+00 2 0 0 0 0 2 0 0 0 0 0 >>>> MatCUSPCopyTo 1 1.0 2.4664e-01 1.0 0.00e+00 0.0 0.0e+00 >>>> 0.0e+00 0.0e+00 5 0 0 0 0 5 0 0 0 0 0 >>>> VecSet 68 1.0 5.1665e-02 1.0 0.00e+00 0.0 0.0e+00 >>>> 0.0e+00 0.0e+00 1 0 0 0 0 1 0 0 0 0 0 >>>> VecAssemblyBegin 17 1.0 5.2691e-05 1.0 0.00e+00 0.0 0.0e+00 >>>> 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>>> VecAssemblyEnd 17 1.0 4.3631e-05 1.0 0.00e+00 0.0 0.0e+00 >>>> 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>>> VecScatterBegin 15 1.0 1.5345e-03 1.0 0.00e+00 0.0 0.0e+00 >>>> 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>>> VecCUSPCopyFrom 1 1.0 1.1199e-03 1.0 0.00e+00 0.0 0.0e+00 >>>> 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>>> KSPSetUp 1 1.0 5.1929e-02 1.0 0.00e+00 0.0 0.0e+00 >>>> 0.0e+00 0.0e+00 1 0 0 0 0 1 0 0 0 0 0 >>>> PCSetUp 2 1.0 8.6590e-02 1.0 3.49e+07 1.0 0.0e+00 >>>> 0.0e+00 0.0e+00 2100 0 0 0 2100 0 0 0 403 >>>> ------------------------------------------------------------ >>>> ------------------------------------------------------------ >>>> >>>> Memory usage is given in bytes: >>>> >>>> Object Type Creations Destructions Memory Descendants' >>>> Mem. >>>> Reports information only for process 0. >>>> >>>> --- Event Stage 0: Main Stage >>>> >>>> Matrix 3 1 52856972 0. >>>> Matrix Null Space 1 1 608 0. >>>> Vector 66 3 3414600 0. >>>> Vector Scatter 1 1 680 0. >>>> Viewer 3 2 1680 0. >>>> Krylov Solver 1 0 0 0. >>>> Preconditioner 2 1 864 0. >>>> Index Set 4 1 800 0. >>>> ============================================================ >>>> ============================================================ >>>> Average time to get PetscTime(): 9.53674e-08 >>>> #PETSc Option Table entries: >>>> -ksp_view >>>> -log_view >>>> -mat_type aijcusparse >>>> -matload_block_size 1 >>>> -vec_type cusp >>>> #End of PETSc Option Table entries >>>> Compiled without FORTRAN kernels >>>> Compiled with full precision matrices (default) >>>> sizeof(short) 2 sizeof(int) 4 sizeof(long) 8 sizeof(void*) 8 >>>> sizeof(PetscScalar) 8 sizeof(PetscInt) 4 >>>> Configure options: PETSC_ARCH=cuda --with-cc=mpicc --with-cxx=mpic++ >>>> --with-fc=mpifort --COPTFLAGS=-O3 --CXXOPTFLAGS=-O3 --FOPTFLAGS=-O3 >>>> --with-shared-libraries=1 --with-debugging=1 --with-cuda=1 >>>> --with-cuda-arch=sm_60 --with-cusp=1 --with-cusp-dir=/home/valera/cusp >>>> --with-vienacl=1 --download-fblaslapack=1 --download-hypre >>>> ----------------------------------------- >>>> Libraries compiled on Mon Mar 5 16:37:18 2018 on node50 >>>> Machine characteristics: Linux-3.10.0-693.17.1.el7.x86_ >>>> 64-x86_64-with-centos-7.2.1511-Core >>>> Using PETSc directory: /home/valera/petsc >>>> Using PETSc arch: cuda >>>> ----------------------------------------- >>>> >>>> Using C compiler: mpicc -fPIC -Wall -Wwrite-strings >>>> -Wno-strict-aliasing -Wno-unknown-pragmas -fstack-protector >>>> -fvisibility=hidden -O3 >>>> Using Fortran compiler: mpifort -fPIC -Wall -ffree-line-length-0 >>>> -Wno-unused-dummy-argument -O3 >>>> ----------------------------------------- >>>> >>>> Using include paths: -I/home/valera/petsc/cuda/include >>>> -I/home/valera/petsc/include -I/home/valera/petsc/include >>>> -I/home/valera/petsc/cuda/include -I/home/valera/cusp/ >>>> -I/usr/local/cuda/include >>>> ----------------------------------------- >>>> >>>> Using C linker: mpicc >>>> Using Fortran linker: mpifort >>>> Using libraries: -Wl,-rpath,/home/valera/petsc/cuda/lib >>>> -L/home/valera/petsc/cuda/lib -lpetsc >>>> -Wl,-rpath,/home/valera/petsc/cuda/lib >>>> -L/home/valera/petsc/cuda/lib -Wl,-rpath,/usr/local/cuda/lib64 >>>> -L/usr/local/cuda/lib64 -Wl,-rpath,/usr/lib64/openmpi/lib >>>> -L/usr/lib64/openmpi/lib -Wl,-rpath,/usr/lib/gcc/x86_64-redhat-linux/4.8.5 >>>> -L/usr/lib/gcc/x86_64-redhat-linux/4.8.5 -lHYPRE -lflapack -lfblas -lm >>>> -lcufft -lcublas -lcudart -lcusparse -lX11 -lstdc++ -ldl -lmpi_usempi >>>> -lmpi_mpifh -lmpi -lgfortran -lm -lgfortran -lm -lgcc_s -lquadmath >>>> -lpthread -lstdc++ -ldl >>>> ----------------------------------------- >>>> >>>> >>>> >>>> Thanks for your help, >>>> >>>> Manuel >>>> >>>> >>>> >>>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> -- Norbert Wiener >>> >>> https://www.cse.buffalo.edu/~knepley/ <http://www.caam.rice.edu/~mk51/> >>> >> >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ <http://www.caam.rice.edu/~mk51/> >