On Wed, Mar 28, 2012 at 6:42 PM, Matthew Knepley <knepley at gmail.com> wrote:
> On Wed, Mar 28, 2012 at 6:25 PM, David Fuentes <fuentesdt at gmail.com>wrote:
>
>> Yes. This would work.
>> I had trouble compiling in single precision using the some of the
>> external package options I was using for double.
>>
>
> Okay, getting on it now.
>
Okay, I have tested using PetscReal=double with floats on the GPU. If you
want doubles on the GPU, you can
currently change PETSc/FEM.py:434. I am deciding what the best way to
specify this type is.
Thanks,
Matt
> Matt
>
>
>> On Wed, Mar 28, 2012 at 4:57 PM, Matthew Knepley <knepley at gmail.com>wrote:
>>
>>> On Wed, Mar 28, 2012 at 4:12 PM, David Fuentes <fuentesdt at
>>> gmail.com>wrote:
>>>
>>>> works!
>>>>
>>>
>>> Excellent. Now, my thinking was that GPUs are most useful doing single
>>> work, but
>>> I can see the utility of double accuracy for a residual.
>>>
>>> My inclination is to define another type, say GPUReal, and use it for
>>> all kernels.
>>> Would that do what you want?
>>>
>>> Matt
>>>
>>>
>>>> SCRGP2$ make ex52
>>>> /usr/bin/mpicxx -o ex52.o -c -O0 -g -fPIC
>>>> -I/opt/apps/PETSC/petsc-dev/include
>>>> -I/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-single-dbg/include
>>>> -I/opt/apps/cuda/4.0/cuda/include -I/usr/include -I/usr/include/mpich2
>>>> -D__INSDIR__=src/snes/examples/tutorials/ ex52.c
>>>> nvcc -G -O0 -g -arch=sm_10 -c --compiler-options="-O0 -g -fPIC
>>>> -I/opt/apps/PETSC/petsc-dev/include
>>>> -I/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-single-dbg/include
>>>> -I/opt/apps/cuda/4.0/cuda/include -I/usr/include -I/usr/include/mpich2
>>>> -D__INSDIR__=src/snes/examples/tutorials/" ex52_integrateElement.cu
>>>> ex52_gpu_inline.h(7): warning: variable "points_0" was declared but
>>>> never referenced
>>>>
>>>> ex52_gpu_inline.h(21): warning: variable "Basis_0" was declared but
>>>> never referenced
>>>>
>>>> ex52_gpu_inline.h(7): warning: variable "points_0" was declared but
>>>> never referenced
>>>>
>>>> ex52_gpu_inline.h(13): warning: variable "weights_0" was declared but
>>>> never referenced
>>>>
>>>> ex52_gpu_inline.h(21): warning: variable "Basis_0" was declared but
>>>> never referenced
>>>>
>>>> ex52_gpu_inline.h(28): warning: variable "BasisDerivatives_0" was
>>>> declared but never referenced
>>>>
>>>> ex52_gpu_inline.h(7): warning: variable "points_0" was declared but
>>>> never referenced
>>>>
>>>> ex52_gpu_inline.h(21): warning: variable "Basis_0" was declared but
>>>> never referenced
>>>>
>>>> ex52_gpu_inline.h(7): warning: variable "points_0" was declared but
>>>> never referenced
>>>>
>>>> ex52_gpu_inline.h(13): warning: variable "weights_0" was declared but
>>>> never referenced
>>>>
>>>> ex52_gpu_inline.h(21): warning: variable "Basis_0" was declared but
>>>> never referenced
>>>>
>>>> ex52_gpu_inline.h(28): warning: variable "BasisDerivatives_0" was
>>>> declared but never referenced
>>>>
>>>> /usr/bin/mpicxx -O0 -g -o ex52 ex52.o ex52_integrateElement.o
>>>>
>>>> -Wl,-rpath,/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-single-dbg/lib
>>>> -L/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-single-dbg/lib
>>>> -lpetsc
>>>> -Wl,-rpath,/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-single-dbg/lib
>>>> -ltriangle -lX11 -lpthread -lmetis -Wl,-rpath,/opt/apps/cuda/4.0/cuda/lib64
>>>> -L/opt/apps/cuda/4.0/cuda/lib64 -lcufft -lcublas -lcudart -lcusparse
>>>> -Wl,-rpath,/opt/epd-7.1-2-rh5-x86_64/lib -L/opt/epd-7.1-2-rh5-x86_64/lib
>>>> -lmkl_rt -lmkl_intel_thread -lmkl_core -liomp5
>>>> -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/4.4.3
>>>> -L/usr/lib/gcc/x86_64-linux-gnu/4.4.3 -ldl -lmpich -lopa -lpthread -lrt
>>>> -lgcc_s -lmpichf90 -lgfortran -lm -lm -lmpichcxx -lstdc++ -lmpichcxx
>>>> -lstdc++ -ldl -lmpich -lopa -lpthread -lrt -lgcc_s -ldl
>>>>
>>>>
>>>> SCRGP2$ ./ex52 -dim 2 -compute_function -show_residual -batch -gpu
>>>> GPU layout grid(1,2,1) block(3,1,1) with 1 batches
>>>> N_t: 3, N_cb: 1
>>>> Residual:
>>>> Vector Object: 1 MPI processes
>>>> type: seq
>>>> -0.25
>>>> -0.5
>>>> 0.25
>>>> -0.5
>>>> -1
>>>> 0.5
>>>> 0.25
>>>> 0.5
>>>> 0.75
>>>> SCRGP2$ ./ex52 -dim 2 -compute_function -show_residual -batch
>>>> Residual:
>>>> Vector Object: 1 MPI processes
>>>> type: seq
>>>> -0.25
>>>> -0.5
>>>> 0.25
>>>> -0.5
>>>> -1
>>>> 0.5
>>>> 0.25
>>>> 0.5
>>>> 0.75
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On Wed, Mar 28, 2012 at 1:37 PM, David Fuentes <fuentesdt at
>>>> gmail.com>wrote:
>>>>
>>>>> sure. will do.
>>>>>
>>>>>
>>>>> On Wed, Mar 28, 2012 at 1:23 PM, Matthew Knepley <knepley at
>>>>> gmail.com>wrote:
>>>>>
>>>>>> On Wed, Mar 28, 2012 at 1:14 PM, David Fuentes <fuentesdt at
>>>>>> gmail.com>wrote:
>>>>>>
>>>>>>> thanks! its running, but I seem to be getting different answer for
>>>>>>> cpu/gpu ?
>>>>>>> i had some floating point problems on this Tesla M2070 gpu before,
>>>>>>> but adding the '-arch=sm_20' option seemed to fix it last time.
>>>>>>>
>>>>>>>
>>>>>>> is the assembly in single precision ? my 'const PetscReal
>>>>>>> jacobianInverse' being passed in are doubles
>>>>>>>
>>>>>>
>>>>>> Yep, that is the problem. I have not tested anything in double. I
>>>>>> have not decided exactly how to handle it. Can you
>>>>>> make another ARCH --with-precision=single and make sure it works, and
>>>>>> then we can fix the double issue?
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> Matt
>>>>>>
>>>>>>
>>>>>>> SCRGP2$ ./ex52 -dim 2 -compute_function -show_residual -batch -gpu
>>>>>>> GPU layout grid(1,2,1) block(3,1,1) with 1 batches
>>>>>>> N_t: 3, N_cb: 1
>>>>>>> Residual:
>>>>>>> Vector Object: 1 MPI processes
>>>>>>> type: seq
>>>>>>> 0
>>>>>>> 755712
>>>>>>> 0
>>>>>>> -58720
>>>>>>> -2953.13
>>>>>>> 0.375
>>>>>>> 1.50323e+07
>>>>>>> 0.875
>>>>>>> 0
>>>>>>> SCRGP2$
>>>>>>> SCRGP2$ ./ex52 -dim 2 -compute_function -show_residual -batch
>>>>>>> Residual:
>>>>>>> Vector Object: 1 MPI processes
>>>>>>> type: seq
>>>>>>> -0.25
>>>>>>> -0.5
>>>>>>> 0.25
>>>>>>> -0.5
>>>>>>> -1
>>>>>>> 0.5
>>>>>>> 0.25
>>>>>>> 0.5
>>>>>>> 0.75
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Wed, Mar 28, 2012 at 11:55 AM, Matthew Knepley <knepley at gmail.com
>>>>>>> > wrote:
>>>>>>>
>>>>>>>> On Wed, Mar 28, 2012 at 11:45 AM, David Fuentes <
>>>>>>>> fuentesdt at gmail.com> wrote:
>>>>>>>>
>>>>>>>>> The example seems to be running on cpu with '-batch' but
>>>>>>>>> i'm getting errors in line 323 with the '-gpu' option
>>>>>>>>>
>>>>>>>>> [0]PETSC ERROR: IntegrateElementBatchGPU() line 323 in
>>>>>>>>> src/snes/examples/tutorials/ex52_integrateElement.cu
>>>>>>>>>
>>>>>>>>> should this possibly be PetscScalar ?
>>>>>>>>>
>>>>>>>>
>>>>>>>> No.
>>>>>>>>
>>>>>>>>
>>>>>>>>> - ierr = cudaMalloc((void**) &d_coefficients, Ne*N_bt *
>>>>>>>>> sizeof(float));CHKERRQ(ierr);
>>>>>>>>> + ierr = cudaMalloc((void**) &d_coefficients, Ne*N_bt *
>>>>>>>>> sizeof(PetscScalar));CHKERRQ(ierr);
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> SCRGP2$ python
>>>>>>>>> $PETSC_DIR/bin/pythonscripts/PetscGenerateFEMQuadrature.py 2 1 1 1
>>>>>>>>> laplacian ex52.h
>>>>>>>>> ['/opt/apps/PETSC/petsc-dev/bin/pythonscripts/PetscGenerateFEMQuadrature.py',
>>>>>>>>> '2', '1', '1', '1', 'laplacian', 'ex52.h']
>>>>>>>>> 2 1 1 1 laplacian
>>>>>>>>> [{(-1.0, -1.0): [(1.0, ())]}, {(1.0, -1.0): [(1.0, ())]}, {(-1.0,
>>>>>>>>> 1.0): [(1.0, ())]}]
>>>>>>>>> {0: {0: [0], 1: [1], 2: [2]}, 1: {0: [], 1: [], 2: []}, 2: {0: []}}
>>>>>>>>> Perm: [0, 1, 2]
>>>>>>>>> Creating /home/fuentes/snestutorial/ex52.h
>>>>>>>>> Creating /home/fuentes/snestutorial/ex52_gpu.h
>>>>>>>>> [{(-1.0, -1.0): [(1.0, ())]}, {(1.0, -1.0): [(1.0, ())]}, {(-1.0,
>>>>>>>>> 1.0): [(1.0, ())]}]
>>>>>>>>> {0: {0: [0], 1: [1], 2: [2]}, 1: {0: [], 1: [], 2: []}, 2: {0: []}}
>>>>>>>>> Perm: [0, 1, 2]
>>>>>>>>> Creating /home/fuentes/snestutorial/ex52_gpu_inline.h
>>>>>>>>>
>>>>>>>>> SCRGP2$ make ex52
>>>>>>>>> /usr/bin/mpicxx -o ex52.o -c -O0 -g -fPIC
>>>>>>>>> -I/opt/apps/PETSC/petsc-dev/include
>>>>>>>>> -I/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-dbg/include
>>>>>>>>> -I/opt/apps/cuda/4.1/cuda/include
>>>>>>>>> -I/opt/apps/PETSC/petsc-dev/include/sieve
>>>>>>>>> -I/opt/MATLAB/R2011a/extern/include -I/usr/include
>>>>>>>>> -I/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-dbg/cbind/include
>>>>>>>>> -I/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-dbg/forbind/include
>>>>>>>>> -I/usr/include/mpich2 -D__INSDIR__=src/snes/examples/tutorials/ ex52.c
>>>>>>>>> nvcc -O0 -g -arch=sm_20 -c --compiler-options="-O0 -g -fPIC
>>>>>>>>> -I/opt/apps/PETSC/petsc-dev/include
>>>>>>>>> -I/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-dbg/include
>>>>>>>>> -I/opt/apps/cuda/4.1/cuda/include
>>>>>>>>> -I/opt/apps/PETSC/petsc-dev/include/sieve
>>>>>>>>> -I/opt/MATLAB/R2011a/extern/include -I/usr/include
>>>>>>>>> -I/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-dbg/cbind/include
>>>>>>>>> -I/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-dbg/forbind/include
>>>>>>>>> -I/usr/include/mpich2 -D__INSDIR__=src/snes/examples/tutorials/"
>>>>>>>>> ex52_integrateElement.cu
>>>>>>>>> ex52_gpu_inline.h(7): warning: variable "points_0" was declared
>>>>>>>>> but never referenced
>>>>>>>>>
>>>>>>>>> ex52_gpu_inline.h(21): warning: variable "Basis_0" was declared
>>>>>>>>> but never referenced
>>>>>>>>>
>>>>>>>>> ex52_gpu_inline.h(7): warning: variable "points_0" was declared
>>>>>>>>> but never referenced
>>>>>>>>>
>>>>>>>>> ex52_gpu_inline.h(13): warning: variable "weights_0" was declared
>>>>>>>>> but never referenced
>>>>>>>>>
>>>>>>>>> ex52_gpu_inline.h(21): warning: variable "Basis_0" was declared
>>>>>>>>> but never referenced
>>>>>>>>>
>>>>>>>>> ex52_gpu_inline.h(28): warning: variable "BasisDerivatives_0" was
>>>>>>>>> declared but never referenced
>>>>>>>>>
>>>>>>>>> ex52_gpu_inline.h(7): warning: variable "points_0" was declared
>>>>>>>>> but never referenced
>>>>>>>>>
>>>>>>>>> ex52_gpu_inline.h(21): warning: variable "Basis_0" was declared
>>>>>>>>> but never referenced
>>>>>>>>>
>>>>>>>>> ex52_gpu_inline.h(7): warning: variable "points_0" was declared
>>>>>>>>> but never referenced
>>>>>>>>>
>>>>>>>>> ex52_gpu_inline.h(13): warning: variable "weights_0" was declared
>>>>>>>>> but never referenced
>>>>>>>>>
>>>>>>>>> ex52_gpu_inline.h(21): warning: variable "Basis_0" was declared
>>>>>>>>> but never referenced
>>>>>>>>>
>>>>>>>>> ex52_gpu_inline.h(28): warning: variable "BasisDerivatives_0" was
>>>>>>>>> declared but never referenced
>>>>>>>>>
>>>>>>>>> /usr/bin/mpicxx -O0 -g -o ex52 ex52.o ex52_integrateElement.o
>>>>>>>>>
>>>>>>>>> -Wl,-rpath,/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-dbg/lib
>>>>>>>>> -L/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-dbg/lib
>>>>>>>>> -lpetsc
>>>>>>>>> -Wl,-rpath,/opt/apps/PETSC/petsc-dev/gcc-4.4.3-mpich2-1.2-epd-sm_20-dbg/lib
>>>>>>>>> -ltriangle -lX11 -lpthread -lsuperlu_dist_3.0 -lcmumps -ldmumps
>>>>>>>>> -lsmumps
>>>>>>>>> -lzmumps -lmumps_common -lpord -lparmetis -lmetis -lscalapack -lblacs
>>>>>>>>> -Wl,-rpath,/opt/apps/cuda/4.1/cuda/lib64
>>>>>>>>> -L/opt/apps/cuda/4.1/cuda/lib64
>>>>>>>>> -lcufft -lcublas -lcudart -lcusparse
>>>>>>>>> -Wl,-rpath,/opt/MATLAB/R2011a/sys/os/glnxa64:/opt/MATLAB/R2011a/bin/glnxa64:/opt/MATLAB/R2011a/extern/lib/glnxa64
>>>>>>>>> -L/opt/MATLAB/R2011a/bin/glnxa64
>>>>>>>>> -L/opt/MATLAB/R2011a/extern/lib/glnxa64
>>>>>>>>> -leng -lmex -lmx -lmat -lut -licudata -licui18n -licuuc
>>>>>>>>> -Wl,-rpath,/opt/epd-7.1-2-rh5-x86_64/lib
>>>>>>>>> -L/opt/epd-7.1-2-rh5-x86_64/lib
>>>>>>>>> -lmkl_rt -lmkl_intel_thread -lmkl_core -liomp5 -lexoIIv2for -lexodus
>>>>>>>>> -lnetcdf_c++ -lnetcdf -Wl,-rpath,/usr/lib/gcc/x86_64-linux-gnu/4.4.3
>>>>>>>>> -L/usr/lib/gcc/x86_64-linux-gnu/4.4.3 -ldl -lmpich -lopa -lpthread
>>>>>>>>> -lrt
>>>>>>>>> -lgcc_s -lmpichf90 -lgfortran -lm -lm -lmpichcxx -lstdc++ -lmpichcxx
>>>>>>>>> -lstdc++ -ldl -lmpich -lopa -lpthread -lrt -lgcc_s -ldl
>>>>>>>>> /bin/rm -f ex52.o ex52_integrateElement.o
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> SCRGP2$ ./ex52 -dim 2 -compute_function -show_residual -batch
>>>>>>>>> Residual:
>>>>>>>>> Vector Object: 1 MPI processes
>>>>>>>>> type: seq
>>>>>>>>> -0.25
>>>>>>>>> -0.5
>>>>>>>>> 0.25
>>>>>>>>> -0.5
>>>>>>>>> -1
>>>>>>>>> 0.5
>>>>>>>>> 0.25
>>>>>>>>> 0.5
>>>>>>>>> 0.75
>>>>>>>>> SCRGP2$ ./ex52 -dim 2 -compute_function -show_residual -batch -gpu
>>>>>>>>> [0]PETSC ERROR: IntegrateElementBatchGPU() line 323 in
>>>>>>>>> src/snes/examples/tutorials/ex52_integrateElement.cu
>>>>>>>>> [0]PETSC ERROR: FormFunctionLocalBatch() line 679 in
>>>>>>>>> src/snes/examples/tutorials/ex52.c
>>>>>>>>> [0]PETSC ERROR: SNESDMComplexComputeFunction() line 431 in
>>>>>>>>> src/snes/utils/damgsnes.c
>>>>>>>>> [0]PETSC ERROR: main() line 1021 in
>>>>>>>>> src/snes/examples/tutorials/ex52.c
>>>>>>>>> application called MPI_Abort(MPI_COMM_WORLD, 35) - process 0
>>>>>>>>>
>>>>>>>>
>>>>>>>> This is failing on cudaMalloc(), which means your card is not
>>>>>>>> available for running. Are you trying to run on your laptop?
>>>>>>>> If so, applications like Preview can lock up the GPU. I know of no
>>>>>>>> way to test this in CUDA while running. I just close
>>>>>>>> apps until it runs.
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>>
>>>>>>>> Matt
>>>>>>>>
>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Tue, Mar 27, 2012 at 8:37 PM, Matthew Knepley <
>>>>>>>>> knepley at gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> On Tue, Mar 27, 2012 at 2:10 PM, Blaise Bourdin <bourdin at
>>>>>>>>>> lsu.edu>wrote:
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Mar 27, 2012, at 1:23 PM, Matthew Knepley wrote:
>>>>>>>>>>>
>>>>>>>>>>> On Tue, Mar 27, 2012 at 12:58 PM, David Fuentes <
>>>>>>>>>>> fuentesdt at gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi,
>>>>>>>>>>>>
>>>>>>>>>>>> I had a question about the status of example 52.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> http://petsc.cs.iit.edu/petsc/petsc-dev/file/a8e2f2c19319/src/snes/examples/tutorials/ex52.c
>>>>>>>>>>>>
>>>>>>>>>>>> http://petsc.cs.iit.edu/petsc/petsc-dev/file/a8e2f2c19319/src/snes/examples/tutorials/ex52_integrateElement.cu
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Can this example be used with a DM object created from an
>>>>>>>>>>>> unstructured exodusII mesh, DMMeshCreateExodus, And the FEM
>>>>>>>>>>>> assembly done
>>>>>>>>>>>> on GPU ?
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> 1) I have pushed many more tests for it now. They can be run
>>>>>>>>>>> using the Python build system
>>>>>>>>>>>
>>>>>>>>>>> ./config/builder2.py check src/snes/examples/tutorials/ex52.c
>>>>>>>>>>>
>>>>>>>>>>> in fact, you can build any set of files this way.
>>>>>>>>>>>
>>>>>>>>>>> 2) The Exodus creation has to be converted to DMComplex from
>>>>>>>>>>> DMMesh. That should not take me very long. Blaise maintains that
>>>>>>>>>>> so maybe there will be help :) You will just replace
>>>>>>>>>>> DMComplexCreateBoxMesh() with DMComplexCreateExodus(). If you
>>>>>>>>>>> request
>>>>>>>>>>> it, I will bump it up the list.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> DMMeshCreateExodusNG is much more flexible
>>>>>>>>>>> than DMMeshCreateExodus in that it can read meshes with multiple
>>>>>>>>>>> element
>>>>>>>>>>> types and should have a much lower memory footprint. The code
>>>>>>>>>>> should be
>>>>>>>>>>> fairly easy to read. you can email me directly if you have specific
>>>>>>>>>>> questions. I had looked at creating a DMComplex and it did not look
>>>>>>>>>>> too
>>>>>>>>>>> difficult, as long as interpolation is not needed. I have plans to
>>>>>>>>>>> write DMComplexCreateExodus, but haven't had time too so far.
>>>>>>>>>>> Updating the
>>>>>>>>>>> Vec viewers and readers may be a bit more involved. In perfect
>>>>>>>>>>> world, one
>>>>>>>>>>> would write an EXODUS viewer following the lines of the VTK and
>>>>>>>>>>> HDF5 ones.
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> David and Blaise, I have converted this function, now
>>>>>>>>>> DMComplexCreateExodus(). Its not tested, but I think
>>>>>>>>>> Blaise has some stuff we can use to test it.
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>>
>>>>>>>>>> Matt
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> Blaise
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Let me know if you can run the tests.
>>>>>>>>>>>
>>>>>>>>>>> Thanks
>>>>>>>>>>>
>>>>>>>>>>> Matt
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>> Thanks,
>>>>>>>>>>>> David
>>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> What most experimenters take for granted before they begin their
>>>>>>>>>>> experiments is infinitely more interesting than any results to
>>>>>>>>>>> which their
>>>>>>>>>>> experiments lead.
>>>>>>>>>>> -- Norbert Wiener
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Department of Mathematics and Center for Computation & Technology
>>>>>>>>>>> Louisiana State University, Baton Rouge, LA 70803, USA
>>>>>>>>>>> Tel. +1 (225) 578 1612, Fax +1 (225) 578 4276
>>>>>>>>>>> http://www.math.lsu.edu/~bourdin
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> What most experimenters take for granted before they begin their
>>>>>>>>>> experiments is infinitely more interesting than any results to which
>>>>>>>>>> their
>>>>>>>>>> experiments lead.
>>>>>>>>>> -- Norbert Wiener
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> What most experimenters take for granted before they begin their
>>>>>>>> experiments is infinitely more interesting than any results to which
>>>>>>>> their
>>>>>>>> experiments lead.
>>>>>>>> -- Norbert Wiener
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> What most experimenters take for granted before they begin their
>>>>>> experiments is infinitely more interesting than any results to which
>>>>>> their
>>>>>> experiments lead.
>>>>>> -- Norbert Wiener
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>>
>>> --
>>> What most experimenters take for granted before they begin their
>>> experiments is infinitely more interesting than any results to which their
>>> experiments lead.
>>> -- Norbert Wiener
>>>
>>
>>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
--
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120402/bfdfce0e/attachment-0001.htm>
