Re: [petsc-dev] Error running on Titan with GPUs & GNU

[Mark Adams via petsc-dev]

I did not configure hypre manually, so I guess it is not using GPUs.

On Fri, Nov 2, 2018 at 2:40 PM Smith, Barry F.  wrote:

>
>
> > On Nov 2, 2018, at 1:25 PM, Mark Adams  wrote:
> >
> > And I just tested it with GAMG and it seems fine.  And hypre ran, but it
> is not clear that it used GPUs
>
> Presumably hyper must be configured to use GPUs. Currently the PETSc
> hyper download installer hypre.py doesn't have any options for getting
> hypre built for GPUs.
>
> Barry
>
> >
> > 14:13 master= ~/petsc/src/snes/examples/tutorials$ jsrun -n 1 ./ex19
> -dm_vec_type cuda -dm_mat_type aijcusparse -pc_type hypre -ksp_type fgmres
> -snes_monitor_short -snes_rtol 1.e-5 -ksp_view
> > lid velocity = 0.0625, prandtl # = 1., grashof # = 1.
> >   0 SNES Function norm 0.239155
> > KSP Object: 1 MPI processes
> >   type: fgmres
> > restart=30, using Classical (unmodified) Gram-Schmidt
> Orthogonalization with no iterative refinement
> > happy breakdown tolerance 1e-30
> >   maximum iterations=1, initial guess is zero
> >   tolerances:  relative=1e-05, absolute=1e-50, divergence=1.
> >   right preconditioning
> >   using UNPRECONDITIONED norm type for convergence test
> > PC Object: 1 MPI processes
> >   type: hypre
> > HYPRE BoomerAMG preconditioning
> >   Cycle type V
> >   Maximum number of levels 25
> >   Maximum number of iterations PER hypre call 1
> >   Convergence tolerance PER hypre call 0.
> >   Threshold for strong coupling 0.25
> >   Interpolation truncation factor 0.
> >   Interpolation: max elements per row 0
> >   Number of levels of aggressive coarsening 0
> >   Number of paths for aggressive coarsening 1
> >   Maximum row sums 0.9
> >   Sweeps down 1
> >   Sweeps up   1
> >   Sweeps on coarse1
> >   Relax down  symmetric-SOR/Jacobi
> >   Relax upsymmetric-SOR/Jacobi
> >   Relax on coarse Gaussian-elimination
> >   Relax weight  (all)  1.
> >   Outer relax weight (all) 1.
> >   Using CF-relaxation
> >   Not using more complex smoothers.
> >   Measure typelocal
> >   Coarsen typeFalgout
> >   Interpolation type  classical
> >   linear system matrix = precond matrix:
> >   Mat Object: 1 MPI processes
> > type: seqaijcusparse
> > rows=64, cols=64, bs=4
> > total: nonzeros=1024, allocated nonzeros=1024
> > total number of mallocs used during MatSetValues calls =0
> >   using I-node routines: found 16 nodes, limit used is 5
> >   1 SNES Function norm 6.80716e-05
> > KSP Object: 1 MPI processes
> >   type: fgmres
> > restart=30, using Classical (unmodified) Gram-Schmidt
> Orthogonalization with no iterative refinement
> > happy breakdown tolerance 1e-30
> >   maximum iterations=1, initial guess is zero
> >   tolerances:  relative=1e-05, absolute=1e-50, divergence=1.
> >   right preconditioning
> >   using UNPRECONDITIONED norm type for convergence test
> > PC Object: 1 MPI processes
> >   type: hypre
> > HYPRE BoomerAMG preconditioning
> >   Cycle type V
> >   Maximum number of levels 25
> >   Maximum number of iterations PER hypre call 1
> >   Convergence tolerance PER hypre call 0.
> >   Threshold for strong coupling 0.25
> >   Interpolation truncation factor 0.
> >   Interpolation: max elements per row 0
> >   Number of levels of aggressive coarsening 0
> >   Number of paths for aggressive coarsening 1
> >   Maximum row sums 0.9
> >   Sweeps down 1
> >   Sweeps up   1
> >   Sweeps on coarse1
> >   Relax down  symmetric-SOR/Jacobi
> >   Relax upsymmetric-SOR/Jacobi
> >   Relax on coarse Gaussian-elimination
> >   Relax weight  (all)  1.
> >   Outer relax weight (all) 1.
> >   Using CF-relaxation
> >   Not using more complex smoothers.
> >   Measure typelocal
> >   Coarsen typeFalgout
> >   Interpolation type  classical
> >   linear system matrix = precond matrix:
> >   Mat Object: 1 MPI processes
> > type: seqaijcusparse
> > rows=64, cols=64, bs=4
> > total: nonzeros=1024, allocated nonzeros=1024
> > total number of mallocs used during MatSetValues calls =0
> >   using I-node routines: found 16 nodes, limit used is 5
> >   2 SNES Function norm 4.093e-11
> > Number of SNES iterations = 2
> >
> >
> > On Fri, Nov 2, 2018 at 2:10 PM Smith, Barry F. 
> wrote:
> >
> >
> > > On Nov 2, 2018, at 1:03 PM, Mark Adams  wrote:
> > >
> > > FYI, I seem to have the new GPU machine at ORNL (summitdev) working
> with GPUs. That is good enough for now.
> > > Thanks,
> >
> >Excellant!
> >
> > >
> > > 14:00 master= ~/petsc/src/snes/examples/tutorials$ jsrun -n 1 ./ex19
> -dm_vec_type cuda -dm_mat_type aijcusparse -pc_type none -ksp_type fgmres
> -snes_monitor_short -snes_rtol 1.e-5 -ksp_view
> > > lid velocity = 0.0625, prandtl # 

Re: [petsc-dev] Error running on Titan with GPUs & GNU

[Smith, Barry F. via petsc-dev]

> On Nov 2, 2018, at 1:25 PM, Mark Adams  wrote:
> 
> And I just tested it with GAMG and it seems fine.  And hypre ran, but it is 
> not clear that it used GPUs

Presumably hyper must be configured to use GPUs. Currently the PETSc hyper 
download installer hypre.py doesn't have any options for getting hypre built 
for GPUs.

Barry

> 
> 14:13 master= ~/petsc/src/snes/examples/tutorials$ jsrun -n 1 ./ex19 
> -dm_vec_type cuda -dm_mat_type aijcusparse -pc_type hypre -ksp_type fgmres 
> -snes_monitor_short -snes_rtol 1.e-5 -ksp_view
> lid velocity = 0.0625, prandtl # = 1., grashof # = 1.
>   0 SNES Function norm 0.239155 
> KSP Object: 1 MPI processes
>   type: fgmres
> restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization 
> with no iterative refinement
> happy breakdown tolerance 1e-30
>   maximum iterations=1, initial guess is zero
>   tolerances:  relative=1e-05, absolute=1e-50, divergence=1.
>   right preconditioning
>   using UNPRECONDITIONED norm type for convergence test
> PC Object: 1 MPI processes
>   type: hypre
> HYPRE BoomerAMG preconditioning
>   Cycle type V
>   Maximum number of levels 25
>   Maximum number of iterations PER hypre call 1
>   Convergence tolerance PER hypre call 0.
>   Threshold for strong coupling 0.25
>   Interpolation truncation factor 0.
>   Interpolation: max elements per row 0
>   Number of levels of aggressive coarsening 0
>   Number of paths for aggressive coarsening 1
>   Maximum row sums 0.9
>   Sweeps down 1
>   Sweeps up   1
>   Sweeps on coarse1
>   Relax down  symmetric-SOR/Jacobi
>   Relax upsymmetric-SOR/Jacobi
>   Relax on coarse Gaussian-elimination
>   Relax weight  (all)  1.
>   Outer relax weight (all) 1.
>   Using CF-relaxation
>   Not using more complex smoothers.
>   Measure typelocal
>   Coarsen typeFalgout
>   Interpolation type  classical
>   linear system matrix = precond matrix:
>   Mat Object: 1 MPI processes
> type: seqaijcusparse
> rows=64, cols=64, bs=4
> total: nonzeros=1024, allocated nonzeros=1024
> total number of mallocs used during MatSetValues calls =0
>   using I-node routines: found 16 nodes, limit used is 5
>   1 SNES Function norm 6.80716e-05 
> KSP Object: 1 MPI processes
>   type: fgmres
> restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization 
> with no iterative refinement
> happy breakdown tolerance 1e-30
>   maximum iterations=1, initial guess is zero
>   tolerances:  relative=1e-05, absolute=1e-50, divergence=1.
>   right preconditioning
>   using UNPRECONDITIONED norm type for convergence test
> PC Object: 1 MPI processes
>   type: hypre
> HYPRE BoomerAMG preconditioning
>   Cycle type V
>   Maximum number of levels 25
>   Maximum number of iterations PER hypre call 1
>   Convergence tolerance PER hypre call 0.
>   Threshold for strong coupling 0.25
>   Interpolation truncation factor 0.
>   Interpolation: max elements per row 0
>   Number of levels of aggressive coarsening 0
>   Number of paths for aggressive coarsening 1
>   Maximum row sums 0.9
>   Sweeps down 1
>   Sweeps up   1
>   Sweeps on coarse1
>   Relax down  symmetric-SOR/Jacobi
>   Relax upsymmetric-SOR/Jacobi
>   Relax on coarse Gaussian-elimination
>   Relax weight  (all)  1.
>   Outer relax weight (all) 1.
>   Using CF-relaxation
>   Not using more complex smoothers.
>   Measure typelocal
>   Coarsen typeFalgout
>   Interpolation type  classical
>   linear system matrix = precond matrix:
>   Mat Object: 1 MPI processes
> type: seqaijcusparse
> rows=64, cols=64, bs=4
> total: nonzeros=1024, allocated nonzeros=1024
> total number of mallocs used during MatSetValues calls =0
>   using I-node routines: found 16 nodes, limit used is 5
>   2 SNES Function norm 4.093e-11 
> Number of SNES iterations = 2
> 
> 
> On Fri, Nov 2, 2018 at 2:10 PM Smith, Barry F.  wrote:
> 
> 
> > On Nov 2, 2018, at 1:03 PM, Mark Adams  wrote:
> > 
> > FYI, I seem to have the new GPU machine at ORNL (summitdev) working with 
> > GPUs. That is good enough for now.
> > Thanks,
> 
>Excellant!
> 
> > 
> > 14:00 master= ~/petsc/src/snes/examples/tutorials$ jsrun -n 1 ./ex19 
> > -dm_vec_type cuda -dm_mat_type aijcusparse -pc_type none -ksp_type fgmres 
> > -snes_monitor_short -snes_rtol 1.e-5 -ksp_view
> > lid velocity = 0.0625, prandtl # = 1., grashof # = 1.
> >   0 SNES Function norm 0.239155 
> > KSP Object: 1 MPI processes
> >   type: fgmres
> > restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization 
> > with no iterative refinement
> > happy breakdown tolerance 1e-30
> >   maximum iterations=1, initial guess is zero
> >   to

Re: [petsc-dev] Error running on Titan with GPUs & GNU

[Mark Adams via petsc-dev]

And I just tested it with GAMG and it seems fine.  And hypre ran, but it is
not clear that it used GPUs

14:13 master= ~/petsc/src/snes/examples/tutorials$ jsrun -n 1 ./ex19
-dm_vec_type cuda -dm_mat_type aijcusparse -pc_type hypre -ksp_type fgmres
-snes_monitor_short -snes_rtol 1.e-5 -ksp_view
lid velocity = 0.0625, prandtl # = 1., grashof # = 1.
  0 SNES Function norm 0.239155
KSP Object: 1 MPI processes
  type: fgmres
restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization
with no iterative refinement
happy breakdown tolerance 1e-30
  maximum iterations=1, initial guess is zero
  tolerances:  relative=1e-05, absolute=1e-50, divergence=1.
  right preconditioning
  using UNPRECONDITIONED norm type for convergence test
PC Object: 1 MPI processes
  type: hypre
HYPRE BoomerAMG preconditioning
  Cycle type V
  Maximum number of levels 25
  Maximum number of iterations PER hypre call 1
  Convergence tolerance PER hypre call 0.
  Threshold for strong coupling 0.25
  Interpolation truncation factor 0.
  Interpolation: max elements per row 0
  Number of levels of aggressive coarsening 0
  Number of paths for aggressive coarsening 1
  Maximum row sums 0.9
  Sweeps down 1
  Sweeps up   1
  Sweeps on coarse1
  Relax down  symmetric-SOR/Jacobi
  Relax upsymmetric-SOR/Jacobi
  Relax on coarse Gaussian-elimination
  Relax weight  (all)  1.
  Outer relax weight (all) 1.
  Using CF-relaxation
  Not using more complex smoothers.
  Measure typelocal
  Coarsen typeFalgout
  Interpolation type  classical
  linear system matrix = precond matrix:
  Mat Object: 1 MPI processes
type: seqaijcusparse
rows=64, cols=64, bs=4
total: nonzeros=1024, allocated nonzeros=1024
total number of mallocs used during MatSetValues calls =0
  using I-node routines: found 16 nodes, limit used is 5
  1 SNES Function norm 6.80716e-05
KSP Object: 1 MPI processes
  type: fgmres
restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization
with no iterative refinement
happy breakdown tolerance 1e-30
  maximum iterations=1, initial guess is zero
  tolerances:  relative=1e-05, absolute=1e-50, divergence=1.
  right preconditioning
  using UNPRECONDITIONED norm type for convergence test
PC Object: 1 MPI processes
  type: hypre
HYPRE BoomerAMG preconditioning
  Cycle type V
  Maximum number of levels 25
  Maximum number of iterations PER hypre call 1
  Convergence tolerance PER hypre call 0.
  Threshold for strong coupling 0.25
  Interpolation truncation factor 0.
  Interpolation: max elements per row 0
  Number of levels of aggressive coarsening 0
  Number of paths for aggressive coarsening 1
  Maximum row sums 0.9
  Sweeps down 1
  Sweeps up   1
  Sweeps on coarse1
  Relax down  symmetric-SOR/Jacobi
  Relax upsymmetric-SOR/Jacobi
  Relax on coarse Gaussian-elimination
  Relax weight  (all)  1.
  Outer relax weight (all) 1.
  Using CF-relaxation
  Not using more complex smoothers.
  Measure typelocal
  Coarsen typeFalgout
  Interpolation type  classical
  linear system matrix = precond matrix:
  Mat Object: 1 MPI processes
type: seqaijcusparse
rows=64, cols=64, bs=4
total: nonzeros=1024, allocated nonzeros=1024
total number of mallocs used during MatSetValues calls =0
  using I-node routines: found 16 nodes, limit used is 5
  2 SNES Function norm 4.093e-11
Number of SNES iterations = 2


On Fri, Nov 2, 2018 at 2:10 PM Smith, Barry F.  wrote:

>
>
> > On Nov 2, 2018, at 1:03 PM, Mark Adams  wrote:
> >
> > FYI, I seem to have the new GPU machine at ORNL (summitdev) working with
> GPUs. That is good enough for now.
> > Thanks,
>
>Excellant!
>
> >
> > 14:00 master= ~/petsc/src/snes/examples/tutorials$ jsrun -n 1 ./ex19
> -dm_vec_type cuda -dm_mat_type aijcusparse -pc_type none -ksp_type fgmres
> -snes_monitor_short -snes_rtol 1.e-5 -ksp_view
> > lid velocity = 0.0625, prandtl # = 1., grashof # = 1.
> >   0 SNES Function norm 0.239155
> > KSP Object: 1 MPI processes
> >   type: fgmres
> > restart=30, using Classical (unmodified) Gram-Schmidt
> Orthogonalization with no iterative refinement
> > happy breakdown tolerance 1e-30
> >   maximum iterations=1, initial guess is zero
> >   tolerances:  relative=1e-05, absolute=1e-50, divergence=1.
> >   right preconditioning
> >   using UNPRECONDITIONED norm type for convergence test
> > PC Object: 1 MPI processes
> >   type: none
> >   linear system matrix = precond matrix:
> >   Mat Object: 1 MPI processes
> > type: seqaijcusparse
> > rows=64, cols=64, bs=4
> > total: nonzeros=1024, allocated nonzeros=1024
> > total number of mallocs used during MatSetValues calls =0
> 

Re: [petsc-dev] Error running on Titan with GPUs & GNU

[Smith, Barry F. via petsc-dev]

> On Nov 2, 2018, at 1:03 PM, Mark Adams  wrote:
> 
> FYI, I seem to have the new GPU machine at ORNL (summitdev) working with 
> GPUs. That is good enough for now.
> Thanks,

   Excellant!

> 
> 14:00 master= ~/petsc/src/snes/examples/tutorials$ jsrun -n 1 ./ex19 
> -dm_vec_type cuda -dm_mat_type aijcusparse -pc_type none -ksp_type fgmres 
> -snes_monitor_short -snes_rtol 1.e-5 -ksp_view
> lid velocity = 0.0625, prandtl # = 1., grashof # = 1.
>   0 SNES Function norm 0.239155 
> KSP Object: 1 MPI processes
>   type: fgmres
> restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization 
> with no iterative refinement
> happy breakdown tolerance 1e-30
>   maximum iterations=1, initial guess is zero
>   tolerances:  relative=1e-05, absolute=1e-50, divergence=1.
>   right preconditioning
>   using UNPRECONDITIONED norm type for convergence test
> PC Object: 1 MPI processes
>   type: none
>   linear system matrix = precond matrix:
>   Mat Object: 1 MPI processes
> type: seqaijcusparse
> rows=64, cols=64, bs=4
> total: nonzeros=1024, allocated nonzeros=1024
> total number of mallocs used during MatSetValues calls =0
>   using I-node routines: found 16 nodes, limit used is 5
>   1 SNES Function norm 6.82338e-05 
> KSP Object: 1 MPI processes
>   type: fgmres
> restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization 
> with no iterative refinement
> happy breakdown tolerance 1e-30
>   maximum iterations=1, initial guess is zero
>   tolerances:  relative=1e-05, absolute=1e-50, divergence=1.
>   right preconditioning
>   using UNPRECONDITIONED norm type for convergence test
> PC Object: 1 MPI processes
>   type: none
>   linear system matrix = precond matrix:
>   Mat Object: 1 MPI processes
> type: seqaijcusparse
> rows=64, cols=64, bs=4
> total: nonzeros=1024, allocated nonzeros=1024
> total number of mallocs used during MatSetValues calls =0
>   using I-node routines: found 16 nodes, limit used is 5
>   2 SNES Function norm 3.346e-10 
> Number of SNES iterations = 2
> 14:01 master= ~/petsc/src/snes/examples/tutorials$ 
> 
> 
> 
> On Thu, Nov 1, 2018 at 9:33 AM Mark Adams  wrote:
> 
> 
> On Wed, Oct 31, 2018 at 12:30 PM Mark Adams  wrote:
> 
> 
> On Wed, Oct 31, 2018 at 6:59 AM Karl Rupp  wrote:
> Hi Mark,
> 
> ah, I was confused by the Python information at the beginning of 
> configure.log. So it is picking up the correct compiler.
> 
> Have you tried uncommenting the check for GNU?
> 
> Yes, but I am getting an error that the cuda files do not find mpi.h.
>  
> 
> I'm getting a make error.
> 
> Thanks, 

Re: [petsc-dev] Error running on Titan with GPUs & GNU

[Mark Adams via petsc-dev]

FYI, I seem to have the new GPU machine at ORNL (summitdev) working with
GPUs. That is good enough for now.
Thanks,

14:00 master= ~/petsc/src/snes/examples/tutorials$ jsrun -n 1 ./ex19
-dm_vec_type cuda -dm_mat_type aijcusparse -pc_type none -ksp_type fgmres
-snes_monitor_short -snes_rtol 1.e-5 -ksp_view
lid velocity = 0.0625, prandtl # = 1., grashof # = 1.
  0 SNES Function norm 0.239155
KSP Object: 1 MPI processes
  type: fgmres
restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization
with no iterative refinement
happy breakdown tolerance 1e-30
  maximum iterations=1, initial guess is zero
  tolerances:  relative=1e-05, absolute=1e-50, divergence=1.
  right preconditioning
  using UNPRECONDITIONED norm type for convergence test
PC Object: 1 MPI processes
  type: none
  linear system matrix = precond matrix:
  Mat Object: 1 MPI processes
type: seqaijcusparse
rows=64, cols=64, bs=4
total: nonzeros=1024, allocated nonzeros=1024
total number of mallocs used during MatSetValues calls =0
  using I-node routines: found 16 nodes, limit used is 5
  1 SNES Function norm 6.82338e-05
KSP Object: 1 MPI processes
  type: fgmres
restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization
with no iterative refinement
happy breakdown tolerance 1e-30
  maximum iterations=1, initial guess is zero
  tolerances:  relative=1e-05, absolute=1e-50, divergence=1.
  right preconditioning
  using UNPRECONDITIONED norm type for convergence test
PC Object: 1 MPI processes
  type: none
  linear system matrix = precond matrix:
  Mat Object: 1 MPI processes
type: seqaijcusparse
rows=64, cols=64, bs=4
total: nonzeros=1024, allocated nonzeros=1024
total number of mallocs used during MatSetValues calls =0
  using I-node routines: found 16 nodes, limit used is 5
  2 SNES Function norm 3.346e-10
Number of SNES iterations = 2
14:01 master= ~/petsc/src/snes/examples/tutorials$



On Thu, Nov 1, 2018 at 9:33 AM Mark Adams  wrote:

>
>
> On Wed, Oct 31, 2018 at 12:30 PM Mark Adams  wrote:
>
>>
>>
>> On Wed, Oct 31, 2018 at 6:59 AM Karl Rupp  wrote:
>>
>>> Hi Mark,
>>>
>>> ah, I was confused by the Python information at the beginning of
>>> configure.log. So it is picking up the correct compiler.
>>>
>>> Have you tried uncommenting the check for GNU?
>>>
>>
> Yes, but I am getting an error that the cuda files do not find mpi.h.
>
>
>>
>> I'm getting a make error.
>>
>> Thanks,
>>
>

Re: [petsc-dev] Error running on Titan with GPUs & GNU

[Mark Adams via petsc-dev]

On Wed, Oct 31, 2018 at 12:30 PM Mark Adams  wrote:

>
>
> On Wed, Oct 31, 2018 at 6:59 AM Karl Rupp  wrote:
>
>> Hi Mark,
>>
>> ah, I was confused by the Python information at the beginning of
>> configure.log. So it is picking up the correct compiler.
>>
>> Have you tried uncommenting the check for GNU?
>>
>
Yes, but I am getting an error that the cuda files do not find mpi.h.


>
> I'm getting a make error.
>
> Thanks,
>

Re: [petsc-dev] Error running on Titan with GPUs & GNU

[Karl Rupp via petsc-dev]

Hi Mark,

ah, I was confused by the Python information at the beginning of 
configure.log. So it is picking up the correct compiler.


Have you tried uncommenting the check for GNU?

Best regards,
Karli


On 10/31/18 11:40 AM, Mark Adams wrote:
It looks like configure is not finding the correct cc. It does not seem 
hard to find.


06:37 master= /lustre/atlas/proj-shared/geo127/petsc$ cc --version
gcc (GCC) 6.3.0 20161221 (Cray Inc.)
Copyright (C) 2016 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

06:37 master= /lustre/atlas/proj-shared/geo127/petsc$ which cc
/opt/cray/craype/2.5.13/bin/cc
06:38 master= /lustre/atlas/proj-shared/geo127/petsc$ which gcc
/opt/gcc/6.3.0/bin/gcc


On Wed, Oct 31, 2018 at 6:34 AM Mark Adams > wrote:




On Wed, Oct 31, 2018 at 5:05 AM Karl Rupp mailto:r...@iue.tuwien.ac.at>> wrote:

Hi Mark,

please comment or remove lines 83 and 84 in
   config/BuildSystem/config/packages/cuda.py

Is there a compiler newer than GCC 4.3 available?


You mean 6.3?

06:33  ~$ module avail gcc

-
/opt/modulefiles -
gcc/4.8.1          gcc/4.9.3          gcc/6.1.0 
gcc/6.3.0(default) gcc/7.2.0
gcc/4.8.2          gcc/5.3.0          gcc/6.2.0          gcc/7.1.0 
         gcc/7.3.0



Best regards,
Karli



On 10/31/18 8:15 AM, Mark Adams via petsc-dev wrote:
 > After loading a cuda module ...
 >
 > On Wed, Oct 31, 2018 at 2:58 AM Mark Adams mailto:mfad...@lbl.gov>
 > >> wrote:
 >
 >     I get an error with --with-cuda=1
 >
 >     On Tue, Oct 30, 2018 at 4:44 PM Smith, Barry F.
mailto:bsm...@mcs.anl.gov>
 >     >>
wrote:
 >
 >         --with-cudac=1 should be --with-cuda=1
 >
 >
 >
 >          > On Oct 30, 2018, at 12:35 PM, Smith, Barry F. via
petsc-dev
 >         mailto:petsc-dev@mcs.anl.gov>
>>
wrote:
 >          >
 >          >
 >          >
 >          >> On Oct 29, 2018, at 8:09 PM, Mark Adams
mailto:mfad...@lbl.gov>
 >         >> wrote:
 >          >>
 >          >> And a debug build seems to work:
 >          >
 >          >    Well ok.
 >          >
 >          >    Are there newer versions of the Gnu compiler
for this
 >         system? Are there any other compilers on the system
that would
 >         likely be less buggy? IBM compilers? If this simple code
 >         generates a gross error with optimization who's to
say how many
 >         more subtle bugs may be induced in the library by the
buggy
 >         optimizer (there may be none but IMHO probability
says there
 >         will be others).
 >          >
 >          >    Is there any chance that valgrind runs on this
machine;
 >         you could run the optimized version through it and
see what it says.
 >          >
 >          >   Barry
 >          >
 >          >>
 >          >> 21:04 1 master=
/lustre/atlas/proj-shared/geo127/petsc$ make
 >   
  PETSC_DIR=/lustre/atlas/proj-shared/geo127/petsc_titan_dbg64idx_gnu_cuda

 >         PETSC_ARCH="" test
 >          >> Running test examples to verify correct installation
 >          >> Using
 >   
  PETSC_DIR=/lustre/atlas/proj-shared/geo127/petsc_titan_dbg64idx_gnu_cuda

 >         and PETSC_ARCH=
 >          >> ***Error detected during compile or
 >         link!***
 >          >> See
http://www.mcs.anl.gov/petsc/documentation/faq.html
 >          >>
 >   
  /lustre/atlas/proj-shared/geo127/petsc/src/snes/examples/tutorials

 >         ex19
 >          >>
 >   
  *

 >          >> cc -o ex19.o -c -g
 > 
  -I/lustre/atlas/proj-shared/geo127/petsc_titan_dbg64idx_gnu_cuda/include    `pwd`/ex19.c

 >          >> cc -g  -o ex19 ex19.o
 >   
  -L/lustre/atlas/proj-shared/geo127/petsc_titan_dbg64idx_gnu_cuda/lib
 >   
  -Wl,-rpath,/lustre/atl

Re: [petsc-dev] Error running on Titan with GPUs & GNU

[Mark Adams via petsc-dev]

It looks like configure is not finding the correct cc. It does not seem
hard to find.

06:37 master= /lustre/atlas/proj-shared/geo127/petsc$ cc --version
gcc (GCC) 6.3.0 20161221 (Cray Inc.)
Copyright (C) 2016 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

06:37 master= /lustre/atlas/proj-shared/geo127/petsc$ which cc
/opt/cray/craype/2.5.13/bin/cc
06:38 master= /lustre/atlas/proj-shared/geo127/petsc$ which gcc
/opt/gcc/6.3.0/bin/gcc


On Wed, Oct 31, 2018 at 6:34 AM Mark Adams  wrote:

>
>
> On Wed, Oct 31, 2018 at 5:05 AM Karl Rupp  wrote:
>
>> Hi Mark,
>>
>> please comment or remove lines 83 and 84 in
>>   config/BuildSystem/config/packages/cuda.py
>>
>> Is there a compiler newer than GCC 4.3 available?
>>
>
> You mean 6.3?
>
> 06:33  ~$ module avail gcc
>
> - /opt/modulefiles
> -
> gcc/4.8.1  gcc/4.9.3  gcc/6.1.0
> gcc/6.3.0(default) gcc/7.2.0
> gcc/4.8.2  gcc/5.3.0  gcc/6.2.0  gcc/7.1.0
>   gcc/7.3.0
>
>
>
>>
>> Best regards,
>> Karli
>>
>>
>>
>> On 10/31/18 8:15 AM, Mark Adams via petsc-dev wrote:
>> > After loading a cuda module ...
>> >
>> > On Wed, Oct 31, 2018 at 2:58 AM Mark Adams > > > wrote:
>> >
>> > I get an error with --with-cuda=1
>> >
>> > On Tue, Oct 30, 2018 at 4:44 PM Smith, Barry F. > > > wrote:
>> >
>> > --with-cudac=1 should be --with-cuda=1
>> >
>> >
>> >
>> >  > On Oct 30, 2018, at 12:35 PM, Smith, Barry F. via petsc-dev
>> > mailto:petsc-dev@mcs.anl.gov>> wrote:
>> >  >
>> >  >
>> >  >
>> >  >> On Oct 29, 2018, at 8:09 PM, Mark Adams > > > wrote:
>> >  >>
>> >  >> And a debug build seems to work:
>> >  >
>> >  >Well ok.
>> >  >
>> >  >Are there newer versions of the Gnu compiler for this
>> > system? Are there any other compilers on the system that would
>> > likely be less buggy? IBM compilers? If this simple code
>> > generates a gross error with optimization who's to say how many
>> > more subtle bugs may be induced in the library by the buggy
>> > optimizer (there may be none but IMHO probability says there
>> > will be others).
>> >  >
>> >  >Is there any chance that valgrind runs on this machine;
>> > you could run the optimized version through it and see what it
>> says.
>> >  >
>> >  >   Barry
>> >  >
>> >  >>
>> >  >> 21:04 1 master= /lustre/atlas/proj-shared/geo127/petsc$ make
>> >
>>  PETSC_DIR=/lustre/atlas/proj-shared/geo127/petsc_titan_dbg64idx_gnu_cuda
>> > PETSC_ARCH="" test
>> >  >> Running test examples to verify correct installation
>> >  >> Using
>> >
>>  PETSC_DIR=/lustre/atlas/proj-shared/geo127/petsc_titan_dbg64idx_gnu_cuda
>> > and PETSC_ARCH=
>> >  >> ***Error detected during compile or
>> > link!***
>> >  >> See http://www.mcs.anl.gov/petsc/documentation/faq.html
>> >  >>
>> >
>>  /lustre/atlas/proj-shared/geo127/petsc/src/snes/examples/tutorials
>> > ex19
>> >  >>
>> >
>>  
>> *
>> >  >> cc -o ex19.o -c -g
>> >
>>  -I/lustre/atlas/proj-shared/geo127/petsc_titan_dbg64idx_gnu_cuda/include
>>   `pwd`/ex19.c
>> >  >> cc -g  -o ex19 ex19.o
>> >
>>  -L/lustre/atlas/proj-shared/geo127/petsc_titan_dbg64idx_gnu_cuda/lib
>> >
>>  
>> -Wl,-rpath,/lustre/atlas/proj-shared/geo127/petsc_titan_dbg64idx_gnu_cuda/lib
>> >
>>  -L/lustre/atlas/proj-shared/geo127/petsc_titan_dbg64idx_gnu_cuda/lib
>> > -lpetsc -lHYPRE -lflapack -lfblas -lparmetis -lmetis -ldl
>> >  >>
>> >
>>  
>> /lustre/atlas/proj-shared/geo127/petsc_titan_dbg64idx_gnu_cuda/lib/libpetsc.a(dlimpl.o):
>> > In function `PetscDLOpen':
>> >  >>
>> >
>>  /lustre/atlas1/geo127/proj-shared/petsc/src/sys/dll/dlimpl.c:108: warning:
>> > Using 'dlopen' in statically linked applications requires at
>> > runtime the shared libraries from the glibc version used for
>> linking
>> >  >>
>> >
>>  
>> /lustre/atlas/proj-shared/geo127/petsc_titan_dbg64idx_gnu_cuda/lib/libpetsc.a(send.o):
>> > In function `PetscOpenSocket':
>> >  >>
>> >
>>  
>> /lustre/atlas1/geo127/proj-shared/petsc/src/sys/classes/viewer/impls/socket/send.c:108:
>> > warning: Using 'gethostbyname' in statically linked applications
>> > requires at runtime the shared libraries from the glibc version
>> > used for linking
>> >  >> true ex19
>> >  >> rm

Re: [petsc-dev] Error running on Titan with GPUs & GNU

[Mark Adams via petsc-dev]

On Wed, Oct 31, 2018 at 5:05 AM Karl Rupp  wrote:

> Hi Mark,
>
> please comment or remove lines 83 and 84 in
>   config/BuildSystem/config/packages/cuda.py
>
> Is there a compiler newer than GCC 4.3 available?
>

You mean 6.3?

06:33  ~$ module avail gcc

- /opt/modulefiles
-
gcc/4.8.1  gcc/4.9.3  gcc/6.1.0  gcc/6.3.0(default)
gcc/7.2.0
gcc/4.8.2  gcc/5.3.0  gcc/6.2.0  gcc/7.1.0
gcc/7.3.0



>
> Best regards,
> Karli
>
>
>
> On 10/31/18 8:15 AM, Mark Adams via petsc-dev wrote:
> > After loading a cuda module ...
> >
> > On Wed, Oct 31, 2018 at 2:58 AM Mark Adams  > > wrote:
> >
> > I get an error with --with-cuda=1
> >
> > On Tue, Oct 30, 2018 at 4:44 PM Smith, Barry F.  > > wrote:
> >
> > --with-cudac=1 should be --with-cuda=1
> >
> >
> >
> >  > On Oct 30, 2018, at 12:35 PM, Smith, Barry F. via petsc-dev
> > mailto:petsc-dev@mcs.anl.gov>> wrote:
> >  >
> >  >
> >  >
> >  >> On Oct 29, 2018, at 8:09 PM, Mark Adams  > > wrote:
> >  >>
> >  >> And a debug build seems to work:
> >  >
> >  >Well ok.
> >  >
> >  >Are there newer versions of the Gnu compiler for this
> > system? Are there any other compilers on the system that would
> > likely be less buggy? IBM compilers? If this simple code
> > generates a gross error with optimization who's to say how many
> > more subtle bugs may be induced in the library by the buggy
> > optimizer (there may be none but IMHO probability says there
> > will be others).
> >  >
> >  >Is there any chance that valgrind runs on this machine;
> > you could run the optimized version through it and see what it
> says.
> >  >
> >  >   Barry
> >  >
> >  >>
> >  >> 21:04 1 master= /lustre/atlas/proj-shared/geo127/petsc$ make
> >
>  PETSC_DIR=/lustre/atlas/proj-shared/geo127/petsc_titan_dbg64idx_gnu_cuda
> > PETSC_ARCH="" test
> >  >> Running test examples to verify correct installation
> >  >> Using
> >
>  PETSC_DIR=/lustre/atlas/proj-shared/geo127/petsc_titan_dbg64idx_gnu_cuda
> > and PETSC_ARCH=
> >  >> ***Error detected during compile or
> > link!***
> >  >> See http://www.mcs.anl.gov/petsc/documentation/faq.html
> >  >>
> >
>  /lustre/atlas/proj-shared/geo127/petsc/src/snes/examples/tutorials
> > ex19
> >  >>
> >
>  
> *
> >  >> cc -o ex19.o -c -g
> >
>  -I/lustre/atlas/proj-shared/geo127/petsc_titan_dbg64idx_gnu_cuda/include
>   `pwd`/ex19.c
> >  >> cc -g  -o ex19 ex19.o
> >
>  -L/lustre/atlas/proj-shared/geo127/petsc_titan_dbg64idx_gnu_cuda/lib
> >
>  -Wl,-rpath,/lustre/atlas/proj-shared/geo127/petsc_titan_dbg64idx_gnu_cuda/lib
> >
>  -L/lustre/atlas/proj-shared/geo127/petsc_titan_dbg64idx_gnu_cuda/lib
> > -lpetsc -lHYPRE -lflapack -lfblas -lparmetis -lmetis -ldl
> >  >>
> >
>  
> /lustre/atlas/proj-shared/geo127/petsc_titan_dbg64idx_gnu_cuda/lib/libpetsc.a(dlimpl.o):
> > In function `PetscDLOpen':
> >  >>
> >
>  /lustre/atlas1/geo127/proj-shared/petsc/src/sys/dll/dlimpl.c:108: warning:
> > Using 'dlopen' in statically linked applications requires at
> > runtime the shared libraries from the glibc version used for
> linking
> >  >>
> >
>  
> /lustre/atlas/proj-shared/geo127/petsc_titan_dbg64idx_gnu_cuda/lib/libpetsc.a(send.o):
> > In function `PetscOpenSocket':
> >  >>
> >
>  
> /lustre/atlas1/geo127/proj-shared/petsc/src/sys/classes/viewer/impls/socket/send.c:108:
> > warning: Using 'gethostbyname' in statically linked applications
> > requires at runtime the shared libraries from the glibc version
> > used for linking
> >  >> true ex19
> >  >> rm ex19.o
> >  >> Possible error running C/C++
> > src/snes/examples/tutorials/ex19 with 1 MPI process
> >  >> See http://www.mcs.anl.gov/petsc/documentation/faq.html
> >  >> lid velocity = 0.0016, prandtl # = 1., grashof # = 1.
> >  >> Number of SNES iterations = 2
> >  >> Application 19081049 resources: utime ~1s, stime ~1s, Rss
> > ~17112, inblocks ~36504, outblocks ~111043
> >  >> Possible error running C/C++
> > src/snes/examples/tutorials/ex19 with 2 MPI processes
> >  >> See http://www.mcs.anl.gov/petsc/documentation/faq.html
> >  >> lid velocity = 0.0016, prandtl # = 1., grashof # = 1.
> >  >> Number of SNES iterations = 2
> >  >> Application 19081

Re: [petsc-dev] Error running on Titan with GPUs & GNU

[Karl Rupp via petsc-dev]

Hi Mark,

please comment or remove lines 83 and 84 in
 config/BuildSystem/config/packages/cuda.py

Is there a compiler newer than GCC 4.3 available?

Best regards,
Karli



On 10/31/18 8:15 AM, Mark Adams via petsc-dev wrote:

After loading a cuda module ...

On Wed, Oct 31, 2018 at 2:58 AM Mark Adams > wrote:


I get an error with --with-cuda=1

On Tue, Oct 30, 2018 at 4:44 PM Smith, Barry F. mailto:bsm...@mcs.anl.gov>> wrote:

--with-cudac=1 should be --with-cuda=1



 > On Oct 30, 2018, at 12:35 PM, Smith, Barry F. via petsc-dev
mailto:petsc-dev@mcs.anl.gov>> wrote:
 >
 >
 >
 >> On Oct 29, 2018, at 8:09 PM, Mark Adams mailto:mfad...@lbl.gov>> wrote:
 >>
 >> And a debug build seems to work:
 >
 >    Well ok.
 >
 >    Are there newer versions of the Gnu compiler for this
system? Are there any other compilers on the system that would
likely be less buggy? IBM compilers? If this simple code
generates a gross error with optimization who's to say how many
more subtle bugs may be induced in the library by the buggy
optimizer (there may be none but IMHO probability says there
will be others).
 >
 >    Is there any chance that valgrind runs on this machine;
you could run the optimized version through it and see what it says.
 >
 >   Barry
 >
 >>
 >> 21:04 1 master= /lustre/atlas/proj-shared/geo127/petsc$ make
PETSC_DIR=/lustre/atlas/proj-shared/geo127/petsc_titan_dbg64idx_gnu_cuda
PETSC_ARCH="" test
 >> Running test examples to verify correct installation
 >> Using
PETSC_DIR=/lustre/atlas/proj-shared/geo127/petsc_titan_dbg64idx_gnu_cuda
and PETSC_ARCH=
 >> ***Error detected during compile or
link!***
 >> See http://www.mcs.anl.gov/petsc/documentation/faq.html
 >>
/lustre/atlas/proj-shared/geo127/petsc/src/snes/examples/tutorials
ex19
 >>

*
 >> cc -o ex19.o -c -g 
  -I/lustre/atlas/proj-shared/geo127/petsc_titan_dbg64idx_gnu_cuda/include    `pwd`/ex19.c
 >> cc -g  -o ex19 ex19.o 
-L/lustre/atlas/proj-shared/geo127/petsc_titan_dbg64idx_gnu_cuda/lib


-Wl,-rpath,/lustre/atlas/proj-shared/geo127/petsc_titan_dbg64idx_gnu_cuda/lib
-L/lustre/atlas/proj-shared/geo127/petsc_titan_dbg64idx_gnu_cuda/lib
-lpetsc -lHYPRE -lflapack -lfblas -lparmetis -lmetis -ldl
 >>

/lustre/atlas/proj-shared/geo127/petsc_titan_dbg64idx_gnu_cuda/lib/libpetsc.a(dlimpl.o):
In function `PetscDLOpen':
 >>
/lustre/atlas1/geo127/proj-shared/petsc/src/sys/dll/dlimpl.c:108: 
warning:
Using 'dlopen' in statically linked applications requires at
runtime the shared libraries from the glibc version used for linking
 >>

/lustre/atlas/proj-shared/geo127/petsc_titan_dbg64idx_gnu_cuda/lib/libpetsc.a(send.o):
In function `PetscOpenSocket':
 >>

/lustre/atlas1/geo127/proj-shared/petsc/src/sys/classes/viewer/impls/socket/send.c:108:
warning: Using 'gethostbyname' in statically linked applications
requires at runtime the shared libraries from the glibc version
used for linking
 >> true ex19
 >> rm ex19.o
 >> Possible error running C/C++
src/snes/examples/tutorials/ex19 with 1 MPI process
 >> See http://www.mcs.anl.gov/petsc/documentation/faq.html
 >> lid velocity = 0.0016, prandtl # = 1., grashof # = 1.
 >> Number of SNES iterations = 2
 >> Application 19081049 resources: utime ~1s, stime ~1s, Rss
~17112, inblocks ~36504, outblocks ~111043
 >> Possible error running C/C++
src/snes/examples/tutorials/ex19 with 2 MPI processes
 >> See http://www.mcs.anl.gov/petsc/documentation/faq.html
 >> lid velocity = 0.0016, prandtl # = 1., grashof # = 1.
 >> Number of SNES iterations = 2
 >> Application 19081050 resources: utime ~1s, stime ~1s, Rss
~19816, inblocks ~36527, outblocks ~111043
 >> 5a6
 >>> Application 19081051 resources: utime ~1s, stime ~0s, Rss
~13864, inblocks ~36527, outblocks ~111043
 >>
/lustre/atlas/proj-shared/geo127/petsc/src/snes/examples/tutorials
 >> Possible problem with ex19_hypre, diffs above
 >> =
 >> ***Error detected during compile or
link!***
 >> See http://www.mcs.anl.gov/petsc/documentation/faq.html
 >>
/lustre/atlas/proj-shared/geo127/petsc/src/snes/examples/tutorials
ex5f
 >> **

Re: [petsc-dev] Error running on Titan with GPUs & GNU

[Smith, Barry F. via petsc-dev]

--with-cudac=1 should be --with-cuda=1



> On Oct 30, 2018, at 12:35 PM, Smith, Barry F. via petsc-dev 
>  wrote:
> 
> 
> 
>> On Oct 29, 2018, at 8:09 PM, Mark Adams  wrote:
>> 
>> And a debug build seems to work:
> 
>Well ok. 
> 
>Are there newer versions of the Gnu compiler for this system? Are there 
> any other compilers on the system that would likely be less buggy? IBM 
> compilers? If this simple code generates a gross error with optimization 
> who's to say how many more subtle bugs may be induced in the library by the 
> buggy optimizer (there may be none but IMHO probability says there will be 
> others). 
> 
>Is there any chance that valgrind runs on this machine; you could run the 
> optimized version through it and see what it says.
> 
>   Barry
> 
>> 
>> 21:04 1 master= /lustre/atlas/proj-shared/geo127/petsc$ make 
>> PETSC_DIR=/lustre/atlas/proj-shared/geo127/petsc_titan_dbg64idx_gnu_cuda 
>> PETSC_ARCH="" test
>> Running test examples to verify correct installation
>> Using 
>> PETSC_DIR=/lustre/atlas/proj-shared/geo127/petsc_titan_dbg64idx_gnu_cuda and 
>> PETSC_ARCH=
>> ***Error detected during compile or link!***
>> See http://www.mcs.anl.gov/petsc/documentation/faq.html
>> /lustre/atlas/proj-shared/geo127/petsc/src/snes/examples/tutorials ex19
>> *
>> cc -o ex19.o -c -g   
>> -I/lustre/atlas/proj-shared/geo127/petsc_titan_dbg64idx_gnu_cuda/include
>> `pwd`/ex19.c
>> cc -g  -o ex19 ex19.o  
>> -L/lustre/atlas/proj-shared/geo127/petsc_titan_dbg64idx_gnu_cuda/lib 
>> -Wl,-rpath,/lustre/atlas/proj-shared/geo127/petsc_titan_dbg64idx_gnu_cuda/lib
>>  -L/lustre/atlas/proj-shared/geo127/petsc_titan_dbg64idx_gnu_cuda/lib 
>> -lpetsc -lHYPRE -lflapack -lfblas -lparmetis -lmetis -ldl
>> /lustre/atlas/proj-shared/geo127/petsc_titan_dbg64idx_gnu_cuda/lib/libpetsc.a(dlimpl.o):
>>  In function `PetscDLOpen':
>> /lustre/atlas1/geo127/proj-shared/petsc/src/sys/dll/dlimpl.c:108: warning: 
>> Using 'dlopen' in statically linked applications requires at runtime the 
>> shared libraries from the glibc version used for linking
>> /lustre/atlas/proj-shared/geo127/petsc_titan_dbg64idx_gnu_cuda/lib/libpetsc.a(send.o):
>>  In function `PetscOpenSocket':
>> /lustre/atlas1/geo127/proj-shared/petsc/src/sys/classes/viewer/impls/socket/send.c:108:
>>  warning: Using 'gethostbyname' in statically linked applications requires 
>> at runtime the shared libraries from the glibc version used for linking
>> true ex19
>> rm ex19.o
>> Possible error running C/C++ src/snes/examples/tutorials/ex19 with 1 MPI 
>> process
>> See http://www.mcs.anl.gov/petsc/documentation/faq.html
>> lid velocity = 0.0016, prandtl # = 1., grashof # = 1.
>> Number of SNES iterations = 2
>> Application 19081049 resources: utime ~1s, stime ~1s, Rss ~17112, inblocks 
>> ~36504, outblocks ~111043
>> Possible error running C/C++ src/snes/examples/tutorials/ex19 with 2 MPI 
>> processes
>> See http://www.mcs.anl.gov/petsc/documentation/faq.html
>> lid velocity = 0.0016, prandtl # = 1., grashof # = 1.
>> Number of SNES iterations = 2
>> Application 19081050 resources: utime ~1s, stime ~1s, Rss ~19816, inblocks 
>> ~36527, outblocks ~111043
>> 5a6
>>> Application 19081051 resources: utime ~1s, stime ~0s, Rss ~13864, inblocks 
>>> ~36527, outblocks ~111043
>> /lustre/atlas/proj-shared/geo127/petsc/src/snes/examples/tutorials
>> Possible problem with ex19_hypre, diffs above
>> =
>> ***Error detected during compile or link!***
>> See http://www.mcs.anl.gov/petsc/documentation/faq.html
>> /lustre/atlas/proj-shared/geo127/petsc/src/snes/examples/tutorials ex5f
>> *
>> ftn -c -g
>> -I/lustre/atlas/proj-shared/geo127/petsc_titan_dbg64idx_gnu_cuda/include
>> -o ex5f.o ex5f.F90
>> ftn -g   -o ex5f ex5f.o  
>> -L/lustre/atlas/proj-shared/geo127/petsc_titan_dbg64idx_gnu_cuda/lib 
>> -Wl,-rpath,/lustre/atlas/proj-shared/geo127/petsc_titan_dbg64idx_gnu_cuda/lib
>>  -L/lustre/atlas/proj-shared/geo127/petsc_titan_dbg64idx_gnu_cuda/lib 
>> -lpetsc -lHYPRE -lflapack -lfblas -lparmetis -lmetis -ldl
>> /lustre/atlas/proj-shared/geo127/petsc_titan_dbg64idx_gnu_cuda/lib/libpetsc.a(dlimpl.o):
>>  In function `PetscDLOpen':
>> /lustre/atlas1/geo127/proj-shared/petsc/src/sys/dll/dlimpl.c:108: warning: 
>> Using 'dlopen' in statically linked applications requires at runtime the 
>> shared libraries from the glibc version used for linking
>> /lustre/atlas/proj-shared/geo127/petsc_titan_dbg64idx_gnu_cuda/lib/libpetsc.a(send.o):
>>  In function `PetscOpenSocket':
>> /lustre/atlas1/geo127/proj-shared/petsc/src/sys/classes/viewer/impls/socket/send.c:108:
>>  warning: Using 'gethostbyname' in statically linked applications requires 
>> at runtime the shared libraries from the glibc version used for linking
>> rm ex5f.o
>> Pos

Re: [petsc-dev] Error running on Titan with GPUs & GNU

[Mark Adams via petsc-dev]

>
>
>
> Are there newer versions of the Gnu compiler for this system?


Yes:

--
/opt/modulefiles
--
gcc/4.8.1  gcc/4.8.2  gcc/4.9.3  gcc/5.3.0
gcc/6.1.0  gcc/6.2.0  gcc/6.3.0(default) gcc/7.1.0
gcc/7.2.0  gcc/7.3.0



> Are there any other compilers on the system that would likely be less
> buggy? IBM compilers? If this simple code generates a gross error with
> optimization who's to say how many more subtle bugs may be induced in the
> library by the buggy optimizer (there may be none but IMHO probability says
> there will be others).
>

Let me ask them what they recommend to use with cuda codes.


>
> Is there any chance that valgrind runs on this machine; you could run
> the optimized version through it and see what it says.
>
>
Valgrind works but tons of output and I could not see anything interesting
in there.

And, this test does work with 1 processor!

I think this is only a problem when GMRES is uses as the eigen estimator in
Cheby. GMRES solvers work and Cheby works with -mg_levels_esteig_ksp_type
cg.

Re: [petsc-dev] Error running on Titan with GPUs & GNU

[Smith, Barry F. via petsc-dev]

> On Oct 29, 2018, at 8:09 PM, Mark Adams  wrote:
> 
> And a debug build seems to work:

Well ok. 

Are there newer versions of the Gnu compiler for this system? Are there any 
other compilers on the system that would likely be less buggy? IBM compilers? 
If this simple code generates a gross error with optimization who's to say how 
many more subtle bugs may be induced in the library by the buggy optimizer 
(there may be none but IMHO probability says there will be others). 

Is there any chance that valgrind runs on this machine; you could run the 
optimized version through it and see what it says.

   Barry

> 
> 21:04 1 master= /lustre/atlas/proj-shared/geo127/petsc$ make 
> PETSC_DIR=/lustre/atlas/proj-shared/geo127/petsc_titan_dbg64idx_gnu_cuda 
> PETSC_ARCH="" test
> Running test examples to verify correct installation
> Using 
> PETSC_DIR=/lustre/atlas/proj-shared/geo127/petsc_titan_dbg64idx_gnu_cuda and 
> PETSC_ARCH=
> ***Error detected during compile or link!***
> See http://www.mcs.anl.gov/petsc/documentation/faq.html
> /lustre/atlas/proj-shared/geo127/petsc/src/snes/examples/tutorials ex19
> *
> cc -o ex19.o -c -g   
> -I/lustre/atlas/proj-shared/geo127/petsc_titan_dbg64idx_gnu_cuda/include
> `pwd`/ex19.c
> cc -g  -o ex19 ex19.o  
> -L/lustre/atlas/proj-shared/geo127/petsc_titan_dbg64idx_gnu_cuda/lib 
> -Wl,-rpath,/lustre/atlas/proj-shared/geo127/petsc_titan_dbg64idx_gnu_cuda/lib 
> -L/lustre/atlas/proj-shared/geo127/petsc_titan_dbg64idx_gnu_cuda/lib -lpetsc 
> -lHYPRE -lflapack -lfblas -lparmetis -lmetis -ldl
> /lustre/atlas/proj-shared/geo127/petsc_titan_dbg64idx_gnu_cuda/lib/libpetsc.a(dlimpl.o):
>  In function `PetscDLOpen':
> /lustre/atlas1/geo127/proj-shared/petsc/src/sys/dll/dlimpl.c:108: warning: 
> Using 'dlopen' in statically linked applications requires at runtime the 
> shared libraries from the glibc version used for linking
> /lustre/atlas/proj-shared/geo127/petsc_titan_dbg64idx_gnu_cuda/lib/libpetsc.a(send.o):
>  In function `PetscOpenSocket':
> /lustre/atlas1/geo127/proj-shared/petsc/src/sys/classes/viewer/impls/socket/send.c:108:
>  warning: Using 'gethostbyname' in statically linked applications requires at 
> runtime the shared libraries from the glibc version used for linking
> true ex19
> rm ex19.o
> Possible error running C/C++ src/snes/examples/tutorials/ex19 with 1 MPI 
> process
> See http://www.mcs.anl.gov/petsc/documentation/faq.html
> lid velocity = 0.0016, prandtl # = 1., grashof # = 1.
> Number of SNES iterations = 2
> Application 19081049 resources: utime ~1s, stime ~1s, Rss ~17112, inblocks 
> ~36504, outblocks ~111043
> Possible error running C/C++ src/snes/examples/tutorials/ex19 with 2 MPI 
> processes
> See http://www.mcs.anl.gov/petsc/documentation/faq.html
> lid velocity = 0.0016, prandtl # = 1., grashof # = 1.
> Number of SNES iterations = 2
> Application 19081050 resources: utime ~1s, stime ~1s, Rss ~19816, inblocks 
> ~36527, outblocks ~111043
> 5a6
> > Application 19081051 resources: utime ~1s, stime ~0s, Rss ~13864, inblocks 
> > ~36527, outblocks ~111043
> /lustre/atlas/proj-shared/geo127/petsc/src/snes/examples/tutorials
> Possible problem with ex19_hypre, diffs above
> =
> ***Error detected during compile or link!***
> See http://www.mcs.anl.gov/petsc/documentation/faq.html
> /lustre/atlas/proj-shared/geo127/petsc/src/snes/examples/tutorials ex5f
> *
> ftn -c -g
> -I/lustre/atlas/proj-shared/geo127/petsc_titan_dbg64idx_gnu_cuda/include
> -o ex5f.o ex5f.F90
> ftn -g   -o ex5f ex5f.o  
> -L/lustre/atlas/proj-shared/geo127/petsc_titan_dbg64idx_gnu_cuda/lib 
> -Wl,-rpath,/lustre/atlas/proj-shared/geo127/petsc_titan_dbg64idx_gnu_cuda/lib 
> -L/lustre/atlas/proj-shared/geo127/petsc_titan_dbg64idx_gnu_cuda/lib -lpetsc 
> -lHYPRE -lflapack -lfblas -lparmetis -lmetis -ldl
> /lustre/atlas/proj-shared/geo127/petsc_titan_dbg64idx_gnu_cuda/lib/libpetsc.a(dlimpl.o):
>  In function `PetscDLOpen':
> /lustre/atlas1/geo127/proj-shared/petsc/src/sys/dll/dlimpl.c:108: warning: 
> Using 'dlopen' in statically linked applications requires at runtime the 
> shared libraries from the glibc version used for linking
> /lustre/atlas/proj-shared/geo127/petsc_titan_dbg64idx_gnu_cuda/lib/libpetsc.a(send.o):
>  In function `PetscOpenSocket':
> /lustre/atlas1/geo127/proj-shared/petsc/src/sys/classes/viewer/impls/socket/send.c:108:
>  warning: Using 'gethostbyname' in statically linked applications requires at 
> runtime the shared libraries from the glibc version used for linking
> rm ex5f.o
> Possible error running Fortran example src/snes/examples/tutorials/ex5f with 
> 1 MPI process
> See http://www.mcs.anl.gov/petsc/documentation/faq.html
> Number of SNES iterations = 4
> Application 19081055 resources: utime 

Re: [petsc-dev] Error running on Titan with GPUs & GNU

[Mark Adams via petsc-dev]

And a debug build seems to work:

21:04 1 master= /lustre/atlas/proj-shared/geo127/petsc$ make
PETSC_DIR=/lustre/atlas/proj-shared/geo127/petsc_titan_dbg64idx_gnu_cuda
PETSC_ARCH="" test
Running test examples to verify correct installation
Using
PETSC_DIR=/lustre/atlas/proj-shared/geo127/petsc_titan_dbg64idx_gnu_cuda
and PETSC_ARCH=
***Error detected during compile or link!***
See http://www.mcs.anl.gov/petsc/documentation/faq.html
/lustre/atlas/proj-shared/geo127/petsc/src/snes/examples/tutorials ex19
*
cc -o ex19.o -c -g
 -I/lustre/atlas/proj-shared/geo127/petsc_titan_dbg64idx_gnu_cuda/include
  `pwd`/ex19.c
cc -g  -o ex19 ex19.o
-L/lustre/atlas/proj-shared/geo127/petsc_titan_dbg64idx_gnu_cuda/lib
-Wl,-rpath,/lustre/atlas/proj-shared/geo127/petsc_titan_dbg64idx_gnu_cuda/lib
-L/lustre/atlas/proj-shared/geo127/petsc_titan_dbg64idx_gnu_cuda/lib
-lpetsc -lHYPRE -lflapack -lfblas -lparmetis -lmetis -ldl
/lustre/atlas/proj-shared/geo127/petsc_titan_dbg64idx_gnu_cuda/lib/libpetsc.a(dlimpl.o):
In function `PetscDLOpen':
/lustre/atlas1/geo127/proj-shared/petsc/src/sys/dll/dlimpl.c:108: warning:
Using 'dlopen' in statically linked applications requires at runtime the
shared libraries from the glibc version used for linking
/lustre/atlas/proj-shared/geo127/petsc_titan_dbg64idx_gnu_cuda/lib/libpetsc.a(send.o):
In function `PetscOpenSocket':
/lustre/atlas1/geo127/proj-shared/petsc/src/sys/classes/viewer/impls/socket/send.c:108:
warning: Using 'gethostbyname' in statically linked applications requires
at runtime the shared libraries from the glibc version used for linking
true ex19
rm ex19.o
Possible error running C/C++ src/snes/examples/tutorials/ex19 with 1 MPI
process
See http://www.mcs.anl.gov/petsc/documentation/faq.html
lid velocity = 0.0016, prandtl # = 1., grashof # = 1.
Number of SNES iterations = 2
Application 19081049 resources: utime ~1s, stime ~1s, Rss ~17112, inblocks
~36504, outblocks ~111043
Possible error running C/C++ src/snes/examples/tutorials/ex19 with 2 MPI
processes
See http://www.mcs.anl.gov/petsc/documentation/faq.html
lid velocity = 0.0016, prandtl # = 1., grashof # = 1.
Number of SNES iterations = 2
Application 19081050 resources: utime ~1s, stime ~1s, Rss ~19816, inblocks
~36527, outblocks ~111043
5a6
> Application 19081051 resources: utime ~1s, stime ~0s, Rss ~13864,
inblocks ~36527, outblocks ~111043
/lustre/atlas/proj-shared/geo127/petsc/src/snes/examples/tutorials
Possible problem with ex19_hypre, diffs above
=
***Error detected during compile or link!***
See http://www.mcs.anl.gov/petsc/documentation/faq.html
/lustre/atlas/proj-shared/geo127/petsc/src/snes/examples/tutorials ex5f
*
ftn -c -g
-I/lustre/atlas/proj-shared/geo127/petsc_titan_dbg64idx_gnu_cuda/include
-o ex5f.o ex5f.F90
ftn -g   -o ex5f ex5f.o
-L/lustre/atlas/proj-shared/geo127/petsc_titan_dbg64idx_gnu_cuda/lib
-Wl,-rpath,/lustre/atlas/proj-shared/geo127/petsc_titan_dbg64idx_gnu_cuda/lib
-L/lustre/atlas/proj-shared/geo127/petsc_titan_dbg64idx_gnu_cuda/lib
-lpetsc -lHYPRE -lflapack -lfblas -lparmetis -lmetis -ldl
/lustre/atlas/proj-shared/geo127/petsc_titan_dbg64idx_gnu_cuda/lib/libpetsc.a(dlimpl.o):
In function `PetscDLOpen':
/lustre/atlas1/geo127/proj-shared/petsc/src/sys/dll/dlimpl.c:108: warning:
Using 'dlopen' in statically linked applications requires at runtime the
shared libraries from the glibc version used for linking
/lustre/atlas/proj-shared/geo127/petsc_titan_dbg64idx_gnu_cuda/lib/libpetsc.a(send.o):
In function `PetscOpenSocket':
/lustre/atlas1/geo127/proj-shared/petsc/src/sys/classes/viewer/impls/socket/send.c:108:
warning: Using 'gethostbyname' in statically linked applications requires
at runtime the shared libraries from the glibc version used for linking
rm ex5f.o
Possible error running Fortran example src/snes/examples/tutorials/ex5f
with 1 MPI process
See http://www.mcs.anl.gov/petsc/documentation/faq.html
Number of SNES iterations = 4
Application 19081055 resources: utime ~1s, stime ~0s, Rss ~12760, inblocks
~36800, outblocks ~111983
Completed test examples
21:06 master= /lustre/atlas/proj-shared/geo127/petsc$

Re: [petsc-dev] Error running on Titan with GPUs & GNU

[Mark Adams via petsc-dev]

On Mon, Oct 29, 2018 at 6:55 PM Smith, Barry F.  wrote:

>
>Here is the code
>
>
> PetscStackCallBLAS("LAPACKgeev",LAPACKgeev_("N","N",&bn,R,&bN,realpart,imagpart,&sdummy,&idummy,&sdummy,&idummy,work,&lwork,&lierr));
>   if (lierr) SETERRQ1(PETSC_COMM_SELF,PETSC_ERR_LIB,"Error in LAPACK
> routine %d",(int)lierr);
>
>What is unfathomable is that it prints (int) lierr of 0 but then the if
> () test should not be satisfied.
>
>Do a ./configure with debugging turned on, could be an optimizing
> compiler error.
>

Configuring debug now.

Note, I was able to run ex56 (ksp) which does not use GMRES. This error was
from a GMRES method so maybe this is an isolated problem.


>
>Barry
>
>
> > On Oct 29, 2018, at 3:56 PM, Mark Adams via petsc-dev <
> petsc-dev@mcs.anl.gov> wrote:
> >
> > I get this error running the tests using GPUs. An error in an LAPACK
> routine.
> >
> > 16:50 master= /lustre/atlas/proj-shared/geo127/petsc$ make
> PETSC_DIR=/lustre/atlas/proj-shared/geo127/petsc_titan_opt64idx_gnu_cuda
> PETSC_ARCH="" test
> > Running test examples to verify correct installation
> > Using
> PETSC_DIR=/lustre/atlas/proj-shared/geo127/petsc_titan_opt64idx_gnu_cuda
> and PETSC_ARCH=
> > ***Error detected during compile or
> link!***
> > See http://www.mcs.anl.gov/petsc/documentation/faq.html
> > /lustre/atlas/proj-shared/geo127/petsc/src/snes/examples/tutorials ex19
> >
> *
> > cc -o ex19.o -c -O
>  -I/lustre/atlas/proj-shared/geo127/petsc_titan_opt64idx_gnu_cuda/include
>   `pwd`/ex19.c
> > cc -O  -o ex19 ex19.o
> -L/lustre/atlas/proj-shared/geo127/petsc_titan_opt64idx_gnu_cuda/lib
> -Wl,-rpath,/lustre/atlas/proj-shared/geo127/petsc_titan_opt64idx_gnu_cuda/lib
> -L/lustre/atlas/proj-shared/geo127/petsc_titan_opt64idx_gnu_cuda/lib
> -lpetsc -lHYPRE -lparmetis -lmetis -ldl
> >
> /lustre/atlas/proj-shared/geo127/petsc_titan_opt64idx_gnu_cuda/lib/libpetsc.a(dlimpl.o):
> In function `PetscDLOpen':
> > dlimpl.c:(.text+0x3b): warning: Using 'dlopen' in statically linked
> applications requires at runtime the shared libraries from the glibc
> version used for linking
> >
> /lustre/atlas/proj-shared/geo127/petsc_titan_opt64idx_gnu_cuda/lib/libpetsc.a(send.o):
> In function `PetscOpenSocket':
> > send.c:(.text+0x3be): warning: Using 'gethostbyname' in statically
> linked applications requires at runtime the shared libraries from the glibc
> version used for linking
> > true ex19
> > rm ex19.o
> > Possible error running C/C++ src/snes/examples/tutorials/ex19 with 1 MPI
> process
> > See http://www.mcs.anl.gov/petsc/documentation/faq.html
> > lid velocity = 0.0016, prandtl # = 1., grashof # = 1.
> > Number of SNES iterations = 2
> > Application 19079964 resources: utime ~1s, stime ~1s, Rss ~29412,
> inblocks ~37563, outblocks ~131654
> > Possible error running C/C++ src/snes/examples/tutorials/ex19 with 2 MPI
> processes
> > See http://www.mcs.anl.gov/petsc/documentation/faq.html
> > lid velocity = 0.0016, prandtl # = 1., grashof # = 1.
> > [1]PETSC ERROR: [0]PETSC ERROR: - Error Message
> --
> > - Error Message
> --
> > [1]PETSC ERROR: [0]PETSC ERROR: Error in external library
> > Error in external library
> > [1]PETSC ERROR: [0]PETSC ERROR: Error in LAPACK routine 0
> > Error in LAPACK routine 0
> > [1]PETSC ERROR: [0]PETSC ERROR: See
> http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
> > See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble
> shooting.
> > [1]PETSC ERROR: [0]PETSC ERROR: Petsc Development GIT revision:
> v3.10.2-461-g0ed19bb123  GIT Date: 2018-10-29 13:43:53 +0100
> > Petsc Development GIT revision: v3.10.2-461-g0ed19bb123  GIT Date:
> 2018-10-29 13:43:53 +0100
> > [1]PETSC ERROR: [0]PETSC ERROR: ./ex19 on a  named nid16438 by adams Mon
> Oct 29 16:52:05 2018
> > ./ex19 on a  named nid16438 by adams Mon Oct 29 16:52:05 2018
> > [1]PETSC ERROR: [0]PETSC ERROR: Configure options --with-cudac=1
> --with-batch=0
> --prefix=/lustre/atlas/proj-shared/geo127/petsc_titan_opt64idx_gnu_cuda
> --download-hypre --download-metis --download-parmetis --with-cc=cc
> --with-clib-autodetect=0 --with-cxx=CC --with-cxxlib-autodetect=0
> --with-fc=ftn --with-fortranlib-autodetect=0 --with-shared-libraries=0
> --known-mpi-shared-libraries=1 --with-mpiexec=aprun --with-x=0
> --with-64-bit-indices --with-debugging=0
> PETSC_ARCH=arch-titan-opt64idx-gnu-cuda
> PETSC_DIR=/lustre/atlas/proj-shared/geo127/petsc
> > Configure options --with-cudac=1 --with-batch=0
> --prefix=/lustre/atlas/proj-shared/geo127/petsc_titan_opt64idx_gnu_cuda
> --download-hypre --download-metis --download-parmetis --with-cc=cc
> --with-clib-autodetect=0 --with-cxx=CC --with-cxxlib-autodetect=0
> --with-fc=ftn --with-fortranlib-autodetect=0

Re: [petsc-dev] Error running on Titan with GPUs & GNU

[Smith, Barry F. via petsc-dev]

   Here is the code

  
PetscStackCallBLAS("LAPACKgeev",LAPACKgeev_("N","N",&bn,R,&bN,realpart,imagpart,&sdummy,&idummy,&sdummy,&idummy,work,&lwork,&lierr));
  if (lierr) SETERRQ1(PETSC_COMM_SELF,PETSC_ERR_LIB,"Error in LAPACK routine 
%d",(int)lierr);

   What is unfathomable is that it prints (int) lierr of 0 but then the if () 
test should not be satisfied. 

   Do a ./configure with debugging turned on, could be an optimizing compiler 
error.

   Barry


> On Oct 29, 2018, at 3:56 PM, Mark Adams via petsc-dev  
> wrote:
> 
> I get this error running the tests using GPUs. An error in an LAPACK routine.
> 
> 16:50 master= /lustre/atlas/proj-shared/geo127/petsc$ make 
> PETSC_DIR=/lustre/atlas/proj-shared/geo127/petsc_titan_opt64idx_gnu_cuda 
> PETSC_ARCH="" test
> Running test examples to verify correct installation
> Using 
> PETSC_DIR=/lustre/atlas/proj-shared/geo127/petsc_titan_opt64idx_gnu_cuda and 
> PETSC_ARCH=
> ***Error detected during compile or link!***
> See http://www.mcs.anl.gov/petsc/documentation/faq.html
> /lustre/atlas/proj-shared/geo127/petsc/src/snes/examples/tutorials ex19
> *
> cc -o ex19.o -c -O   
> -I/lustre/atlas/proj-shared/geo127/petsc_titan_opt64idx_gnu_cuda/include
> `pwd`/ex19.c
> cc -O  -o ex19 ex19.o  
> -L/lustre/atlas/proj-shared/geo127/petsc_titan_opt64idx_gnu_cuda/lib 
> -Wl,-rpath,/lustre/atlas/proj-shared/geo127/petsc_titan_opt64idx_gnu_cuda/lib 
> -L/lustre/atlas/proj-shared/geo127/petsc_titan_opt64idx_gnu_cuda/lib -lpetsc 
> -lHYPRE -lparmetis -lmetis -ldl
> /lustre/atlas/proj-shared/geo127/petsc_titan_opt64idx_gnu_cuda/lib/libpetsc.a(dlimpl.o):
>  In function `PetscDLOpen':
> dlimpl.c:(.text+0x3b): warning: Using 'dlopen' in statically linked 
> applications requires at runtime the shared libraries from the glibc version 
> used for linking
> /lustre/atlas/proj-shared/geo127/petsc_titan_opt64idx_gnu_cuda/lib/libpetsc.a(send.o):
>  In function `PetscOpenSocket':
> send.c:(.text+0x3be): warning: Using 'gethostbyname' in statically linked 
> applications requires at runtime the shared libraries from the glibc version 
> used for linking
> true ex19
> rm ex19.o
> Possible error running C/C++ src/snes/examples/tutorials/ex19 with 1 MPI 
> process
> See http://www.mcs.anl.gov/petsc/documentation/faq.html
> lid velocity = 0.0016, prandtl # = 1., grashof # = 1.
> Number of SNES iterations = 2
> Application 19079964 resources: utime ~1s, stime ~1s, Rss ~29412, inblocks 
> ~37563, outblocks ~131654
> Possible error running C/C++ src/snes/examples/tutorials/ex19 with 2 MPI 
> processes
> See http://www.mcs.anl.gov/petsc/documentation/faq.html
> lid velocity = 0.0016, prandtl # = 1., grashof # = 1.
> [1]PETSC ERROR: [0]PETSC ERROR: - Error Message 
> --
> - Error Message 
> --
> [1]PETSC ERROR: [0]PETSC ERROR: Error in external library
> Error in external library
> [1]PETSC ERROR: [0]PETSC ERROR: Error in LAPACK routine 0
> Error in LAPACK routine 0
> [1]PETSC ERROR: [0]PETSC ERROR: See 
> http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
> See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
> [1]PETSC ERROR: [0]PETSC ERROR: Petsc Development GIT revision: 
> v3.10.2-461-g0ed19bb123  GIT Date: 2018-10-29 13:43:53 +0100
> Petsc Development GIT revision: v3.10.2-461-g0ed19bb123  GIT Date: 2018-10-29 
> 13:43:53 +0100
> [1]PETSC ERROR: [0]PETSC ERROR: ./ex19 on a  named nid16438 by adams Mon Oct 
> 29 16:52:05 2018
> ./ex19 on a  named nid16438 by adams Mon Oct 29 16:52:05 2018
> [1]PETSC ERROR: [0]PETSC ERROR: Configure options --with-cudac=1 
> --with-batch=0 
> --prefix=/lustre/atlas/proj-shared/geo127/petsc_titan_opt64idx_gnu_cuda 
> --download-hypre --download-metis --download-parmetis --with-cc=cc 
> --with-clib-autodetect=0 --with-cxx=CC --with-cxxlib-autodetect=0 
> --with-fc=ftn --with-fortranlib-autodetect=0 --with-shared-libraries=0 
> --known-mpi-shared-libraries=1 --with-mpiexec=aprun --with-x=0 
> --with-64-bit-indices --with-debugging=0 
> PETSC_ARCH=arch-titan-opt64idx-gnu-cuda 
> PETSC_DIR=/lustre/atlas/proj-shared/geo127/petsc
> Configure options --with-cudac=1 --with-batch=0 
> --prefix=/lustre/atlas/proj-shared/geo127/petsc_titan_opt64idx_gnu_cuda 
> --download-hypre --download-metis --download-parmetis --with-cc=cc 
> --with-clib-autodetect=0 --with-cxx=CC --with-cxxlib-autodetect=0 
> --with-fc=ftn --with-fortranlib-autodetect=0 --with-shared-libraries=0 
> --known-mpi-shared-libraries=1 --with-mpiexec=aprun --with-x=0 
> --with-64-bit-indices --with-debugging=0 
> PETSC_ARCH=arch-titan-opt64idx-gnu-cuda 
> PETSC_DIR=/lustre/atlas/proj-shared/geo127/petsc
> [1]PETSC ERROR: [0]PETSC ERROR: #1 KSPComputeEigenvalues_GMRES() line 144 in 
>

Re: [petsc-dev] Error running on Titan with GPUs & GNU

[Mark Adams via petsc-dev]

Still getting this error with downloaded lapack. I sent the logs on the
other thread.


18:02 master= /lustre/atlas/proj-shared/geo127/petsc$ make
PETSC_DIR=/lustre/atlas/proj-shared/geo127/petsc_titan_opt64idx_gnu_cuda
PETSC_ARCH="" test
Running test examples to verify correct installation
Using
PETSC_DIR=/lustre/atlas/proj-shared/geo127/petsc_titan_opt64idx_gnu_cuda
and PETSC_ARCH=
***Error detected during compile or link!***
See http://www.mcs.anl.gov/petsc/documentation/faq.html
/lustre/atlas/proj-shared/geo127/petsc/src/snes/examples/tutorials ex19
*
cc -o ex19.o -c -O
 -I/lustre/atlas/proj-shared/geo127/petsc_titan_opt64idx_gnu_cuda/include
  `pwd`/ex19.c
cc -O  -o ex19 ex19.o
-L/lustre/atlas/proj-shared/geo127/petsc_titan_opt64idx_gnu_cuda/lib
-Wl,-rpath,/lustre/atlas/proj-shared/geo127/petsc_titan_opt64idx_gnu_cuda/lib
-L/lustre/atlas/proj-shared/geo127/petsc_titan_opt64idx_gnu_cuda/lib
-lpetsc -lHYPRE -lflapack -lfblas -lparmetis -lmetis -ldl
/lustre/atlas/proj-shared/geo127/petsc_titan_opt64idx_gnu_cuda/lib/libpetsc.a(dlimpl.o):
In function `PetscDLOpen':
dlimpl.c:(.text+0x3b): warning: Using 'dlopen' in statically linked
applications requires at runtime the shared libraries from the glibc
version used for linking
/lustre/atlas/proj-shared/geo127/petsc_titan_opt64idx_gnu_cuda/lib/libpetsc.a(send.o):
In function `PetscOpenSocket':
send.c:(.text+0x3be): warning: Using 'gethostbyname' in statically linked
applications requires at runtime the shared libraries from the glibc
version used for linking
true ex19
rm ex19.o
Possible error running C/C++ src/snes/examples/tutorials/ex19 with 1 MPI
process
See http://www.mcs.anl.gov/petsc/documentation/faq.html
lid velocity = 0.0016, prandtl # = 1., grashof # = 1.
Number of SNES iterations = 2
Application 19080270 resources: utime ~0s, stime ~1s, Rss ~72056, inblocks
~19397, outblocks ~51049
Possible error running C/C++ src/snes/examples/tutorials/ex19 with 2 MPI
processes
See http://www.mcs.anl.gov/petsc/documentation/faq.html
lid velocity = 0.0016, prandtl # = 1., grashof # = 1.
[1]PETSC ERROR: [0]PETSC ERROR: - Error Message
--
- Error Message
--
[1]PETSC ERROR: [0]PETSC ERROR: Error in external library
Error in external library
[1]PETSC ERROR: [0]PETSC ERROR: Error in LAPACK routine 0
Error in LAPACK routine 0
[1]PETSC ERROR: [0]PETSC ERROR: See
http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble
shooting.
[1]PETSC ERROR: [0]PETSC ERROR: Petsc Development GIT revision:
v3.10.2-461-g0ed19bb123  GIT Date: 2018-10-29 13:43:53 +0100
Petsc Development GIT revision: v3.10.2-461-g0ed19bb123  GIT Date:
2018-10-29 13:43:53 +0100
[1]PETSC ERROR: [0]PETSC ERROR: ./ex19 on a  named nid08331 by adams Mon
Oct 29 18:07:59 2018
./ex19 on a  named nid08331 by adams Mon Oct 29 18:07:59 2018
[1]PETSC ERROR: [0]PETSC ERROR: Configure options --with-cudac=1
--with-batch=0
--prefix=/lustre/atlas/proj-shared/geo127/petsc_titan_opt64idx_gnu_cuda
--download-hypre --download-metis --download-parmetis
--download-fblaslapack --with-cc=cc --with-clib-autodetect=0 --with-cxx=CC
--with-cxxlib-autodetect=0 --with-fc=ftn --with-fortranlib-autodetect=0
--with-shared-libraries=0 --known-mpi-shared-libraries=1
--with-mpiexec=aprun --with-x=0 --with-64-bit-indices --with-debugging=0
PETSC_ARCH=arch-titan-opt64idx-gnu-cuda
PETSC_DIR=/lustre/atlas/proj-shared/geo127/petsc
Configure options --with-cudac=1 --with-batch=0
--prefix=/lustre/atlas/proj-shared/geo127/petsc_titan_opt64idx_gnu_cuda
--download-hypre --download-metis --download-parmetis
--download-fblaslapack --with-cc=cc --with-clib-autodetect=0 --with-cxx=CC
--with-cxxlib-autodetect=0 --with-fc=ftn --with-fortranlib-autodetect=0
--with-shared-libraries=0 --known-mpi-shared-libraries=1
--with-mpiexec=aprun --with-x=0 --with-64-bit-indices --with-debugging=0
PETSC_ARCH=arch-titan-opt64idx-gnu-cuda
PETSC_DIR=/lustre/atlas/proj-shared/geo127/petsc
[1]PETSC ERROR: [0]PETSC ERROR: #1 KSPComputeEigenvalues_GMRES() line 144
in /lustre/atlas1/geo127/proj-shared/petsc/src/ksp/ksp/impls/gmres/gmreig.c
#1 KSPComputeEigenvalues_GMRES() line 144 in
/lustre/atlas1/geo127/proj-shared/petsc/src/ksp/ksp/impls/gmres/gmreig.c
[1]PETSC ERROR: #2 KSPComputeEigenvalues() line 132 in
/lustre/atlas1/geo127/proj-shared/petsc/src/ksp/ksp/interface/itfunc.c
[0]PETSC ERROR: [1]PETSC ERROR: #2 KSPComputeEigenvalues() line 132 in
/lustre/atlas1/geo127/proj-shared/petsc/src/ksp/ksp/interface/itfunc.c
#3 KSPChebyshevComputeExtremeEigenvalues_Private() line 288 in
/lustre/atlas1/geo127/proj-shared/petsc/src/ksp/ksp/impls/cheby/cheby.c
[0]PETSC ERROR: [1]PETSC ERROR: #3
KSPChebyshevComputeExtremeEigenvalues_Private

Re: [petsc-dev] Error running on Titan with GPUs & GNU

[Mark Adams via petsc-dev]

On Mon, Oct 29, 2018 at 5:07 PM Matthew Knepley  wrote:

> On Mon, Oct 29, 2018 at 5:01 PM Mark Adams via petsc-dev <
> petsc-dev@mcs.anl.gov> wrote:
>
>> I get this error running the tests using GPUs. An error in an LAPACK
>> routine.
>>
>
> From the command line, it does not look like GPUs are being used.
>
> It looks like the LAPACK eigensolver is failing. Maybe there is a variant
> signature on this machine?
>

I am not doing anything with LAPACK. I have no ideas what LAPACK it is
picking up. I do notice in the configure log that hypre has LAPACK stuff
embedding in it.  (PETSc is not using hyper's LAPACK is it?).

I can try downloading blaslapack