Re: [petsc-users] GPUs, cud, complex

2019-02-21 Thread Smith, Barry F. via petsc-users 

   Hmm, ex32 suddenly becomes ex39 (and there is no ex39 in the 
src/ksp/ksp/examples/tutorials/ directory?) I try ex32 with those options and 
it runs though the -n1 n2 n3 options aren't used.

   Barry


> On Feb 21, 2019, at 6:20 PM, Randall Mackie  wrote:
> 
> Hi Barry and Satish,
> 
> Yes, sorry, I meant -dm_mat_type_aijcusparse…..
> 
> Here is an attempt to run ex39 under complex:
> 
> 0]PETSC ERROR: 
> 
> [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, 
> probably memory access out of range
> [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
> [0]PETSC ERROR: or see 
> http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
> [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to 
> find memory corruption errors
> [0]PETSC ERROR: likely location of problem given in stack below
> [0]PETSC ERROR: -  Stack Frames 
> 
> [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,
> [0]PETSC ERROR:   INSTEAD the line number of the start of the function
> [0]PETSC ERROR:   is given.
> [0]PETSC ERROR: [0] VecCUDAGetArrayRead line 1283 
> /home/everderio/DEV/petsc-3.10.3/src/vec/vec/impls/seq/seqcuda/veccuda2.cu
> [0]PETSC ERROR: [0] VecAYPX_SeqCUDA line 185 
> /home/everderio/DEV/petsc-3.10.3/src/vec/vec/impls/seq/seqcuda/veccuda2.cu
> [0]PETSC ERROR: [0] VecAYPX line 739 
> /home/everderio/DEV/petsc-3.10.3/src/vec/vec/interface/rvector.c
> [0]PETSC ERROR: [0] KSPBuildResidualDefault line 886 
> /home/everderio/DEV/petsc-3.10.3/src/ksp/ksp/interface/iterativ.c
> [0]PETSC ERROR: [0] KSPBuildResidual line 2132 
> /home/everderio/DEV/petsc-3.10.3/src/ksp/ksp/interface/itfunc.c
> [0]PETSC ERROR: [0] KSPMonitorTrueResidualNorm line 252 
> /home/everderio/DEV/petsc-3.10.3/src/ksp/ksp/interface/iterativ.c
> [0]PETSC ERROR: [0] KSPMonitor line 1714 
> /home/everderio/DEV/petsc-3.10.3/src/ksp/ksp/interface/itfunc.c
> [0]PETSC ERROR: [0] KSPSolve_BCGS line 33 
> /home/everderio/DEV/petsc-3.10.3/src/ksp/ksp/impls/bcgs/bcgs.c
> [0]PETSC ERROR: [0] KSPSolve line 678 
> /home/everderio/DEV/petsc-3.10.3/src/ksp/ksp/interface/itfunc.c
> [0]PETSC ERROR: - Error Message 
> --
> [0]PETSC ERROR: Signal received
> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for 
> trouble shooting.
> [0]PETSC ERROR: Petsc Release Version 3.10.3, Dec, 18, 2018 
> [0]PETSC ERROR: ./ex39_cmplx on a linux-gfortran-complex-debug named GPU by 
> root Thu Feb 21 19:03:37 2019
> [0]PETSC ERROR: Configure options --with-clean=1 --with-scalar-type=complex 
> --with-debugging=1 --with-fortran=1 --with-cuda=1 
> --with-cudac=/usr/local/cuda-10.0/bin/nvcc 
> --download-mpich=./mpich-3.3b1.tar.gz 
> --download-fblaslapack=fblaslapack-3.4.2.tar.gz
> [0]PETSC ERROR: #1 User provided function() line 0 in  unknown file
> application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0
> 
> 
> I used these options:
> 
> #!/bin/bash
> 
> export PETSC_ARCH=linux-gfortran-complex-debug
> 
> ${PETSC_DIR}/lib/petsc/bin/petscmpiexec -n 1 ./ex39_cmplx \
> -ksp_type bcgs \
> -ksp_rtol 1.e-6 \
> -pc_type jacobi \
> -ksp_monitor_true_residual \
> -ksp_converged_reason \
> -mat_type aijcusparse \
> -vec_type cuda \
> -n1 32 \
> -n2 32 \
> -n3 32 \
> 
> 
> My next step was going to try valgrind and see if that turned something up.
> 
> Thanks, Randy
> 
> 
>> On Feb 21, 2019, at 2:51 PM, Smith, Barry F.  wrote:
>> 
>> 
>> Randy,
>> 
>>   Could you please cut and paste the entire error message you get. It worked 
>> for me. 
>> 
>>   I assume you mean -dm_mat_type aijcusparse  not aijcuda (which doesn't 
>> exist).
>> 
>> Satish,
>> 
>> I does appear we do not have a nightly test for cuda and complex, could 
>> that test be added to the nightly sweeps?
>> 
>>  Thanks
>> 
>>  Barry
>> 
>> 
>>> On Feb 14, 2019, at 7:33 PM, Randall Mackie via petsc-users 
>>>  wrote:
>>> 
>>> We are testing whether or not we can benefit from porting our PETSc code to 
>>> use GPUS.
>>> We have installed PETSc following the instructions here:
>>> https://www.mcs.anl.gov/petsc/features/gpus.html
>>> 
>>> Using KSP example 32 (ex32.c) to test, and using -dm_vec_type cuda and 
>>> -dm_mat_type aijcuda 
>>> then ex32 runs fine when compiled with a REAL version of PETSc but bombs 
>>> out when using a COMPLEX version.
>>> 
>>> Is it possible to run PETSc on GPUS in complex mode?d
>>> 
>>> 
>>> Thanks, Randy M.
>> 
>

Re: [petsc-users] GPUs, cud, complex

2019-02-21 Thread Smith, Barry F. via petsc-users 


  Randy,

Could you please cut and paste the entire error message you get. It worked 
for me. 

I assume you mean -dm_mat_type aijcusparse  not aijcuda (which doesn't 
exist).

  Satish,

  I does appear we do not have a nightly test for cuda and complex, could 
that test be added to the nightly sweeps?

   Thanks

   Barry


> On Feb 14, 2019, at 7:33 PM, Randall Mackie via petsc-users 
>  wrote:
> 
> We are testing whether or not we can benefit from porting our PETSc code to 
> use GPUS.
> We have installed PETSc following the instructions here:
> https://www.mcs.anl.gov/petsc/features/gpus.html
> 
> Using KSP example 32 (ex32.c) to test, and using -dm_vec_type cuda and 
> -dm_mat_type aijcuda 
> then ex32 runs fine when compiled with a REAL version of PETSc but bombs out 
> when using a COMPLEX version.
> 
> Is it possible to run PETSc on GPUS in complex mode?d
> 
> 
> Thanks, Randy M.

[petsc-users] Question with filedsplit in PETSc

2019-02-21 Thread Zhu, Qiming via petsc-users 

Dear all,


Sorry to disturb you. I am a user of Petsc. I am trying to use Fieldsplit in 
Petsc to do preconditioning for Navier-Stokes problem. I have some problems 
when I trying to use Fieldsplit function. I am now defining the nest matrix 
first, then I get the IS from nested matrix. But I find that my code just work 
for one core. When I change to parallel case, I could only get zero solution. I 
wonder is there any special requirements for IS definition in Fieldsplit? I 
include one code here. If you have any idea, hope you reply soon. Thank you for 
your help. Thank you very much.


Yours sincerely,

Qiming Zhu,



makefile
Description: makefile
static char help[] = "Test";
#include 
#include 

int main(int argc,char **args)
{
  Mat	Asub[4];
  Mat   Anest;
  Vec	x,b,y;
  IS	isg[2];
  PetscInt row,end,i,j,Ist,Ien;
  PetscErrorCode ierr;
  PetscMPIIntrank,size;
  KSPksp;
  PC pc;
  PetscInt	 kk;

  // Initialize petsc and mpi here
  ierr = PetscInitialize(,,(char*)0,help);if (ierr) return ierr;
  ierr = MPI_Comm_rank(PETSC_COMM_WORLD,);CHKERRQ(ierr);
  ierr = MPI_Comm_size(PETSC_COMM_WORLD,);CHKERRQ(ierr);

  /* My matrix and rhs vec are as followed:
 The matrix Anest I define here is:
  	1 0 0 0 | 0 0 0 0
	0 2 0 0 | 0 0 0 0
	0 0 3 0 | 0 0 0 0
	0 0 0 4 | 0 0 0 0
  	- - - - - - - - -
 	0 0 0 0 | 5 0 0 0
  	0 0 0 0 | 0 6 0 0
	0 0 0 0 | 0 0 7 0
0 0 0 0 | 0 0 0 8

 Thr rhs I define here is :
	1
	2
	3
	4
	-
	5
	6
	7
	8

 So my solution should be all one:
	1
	1
	1
	1
	1
	1
	1
	1
  */
  // kk is submatrix size
  kk=4;
  MatCreate(PETSC_COMM_WORLD,[0]);
  MatSetSizes(Asub[0],PETSC_DECIDE,PETSC_DECIDE,kk,kk);
  MatSetType(Asub[0],MATMPIAIJ);
  MatMPIAIJSetPreallocation(Asub[0],5,NULL,5,NULL);
  MatGetOwnershipRange(Asub[0], , );
  for(i=Ist;i

Re: [petsc-users] About DMDA (and extracting its ordering)

2019-02-21 Thread Thibaut Appel via petsc-users 

Hi Matthew,

Is your first part of your answer (using DMDASetBlockFills) valid only 
in the case I create a DMDA object?


Yes I think that is the kind of stencil I am using. I could know how the 
stencil looks like exactly, but I preallocate looping, for each process 
on all the elements of the stencil, grid node by grid node (which is not 
that costly, and "exact")


If I do NOT use a DMDA object and create my MPIAIJ matrix myself, how do 
I get the global row indices owned by the process (the "DMDA-like" ones 
you mentioned)? The problem is that MatGetOwnershipRange cannot be 
called if the matrix hasn't been preallocated, and I need the global 
indices to preallocate.


Thibaut


On 21/02/2019 17:49, Matthew Knepley wrote:
On Thu, Feb 21, 2019 at 11:16 AM Thibaut Appel via petsc-users 
mailto:petsc-users@mcs.anl.gov>> wrote:


Dear PETSc developers/users,

I’m solving linear PDEs on a regular grid with high-order finite
differences, assembling an MPIAIJ matrix to solve linear systems
or eigenvalue problems. I’ve been using vertex major, natural
ordering for the parallelism with PetscSplitOwnership (yielding
rectangular slices of the physical domain) and wanted to move to
DMDA to have a more square-ish domain decomposition and minimize
communication between processes.

However, my application is memory critical, and I have
finely-tuned matrix preallocation routines for allocating memory
“optimally”. It seems the memory of a DMDA matrix is allocated
along the value of the stencil width of DMDACreate and the manual
says about it

“These DMDA stencils have nothing directly to do with any finite
difference stencils one might chose to use for a discretization”

And despite reading the manual pages there must be something I do
not understand in the DM topology, what is that "stencil width"
for then? I will not use ghost values for my FD-method, right?


What this is saying is, "You might be using some stencil that is not 
STAR or BOX, but we are preallocating according to one of those".
If you really care about how much memory is preallocated, which it 
seems you do, then you might be able to use


https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/DMDA/DMDASetBlockFills.html

to tell use exactly how to preallocate.

I was then wondering if I could just create a MPIAIJ matrix, and
with a PETSc routine get the global indices of the domain for each
process: in other words, an equivalent of PetscSplitOwnership that
gives me the DMDA unknown ordering. So I can feed and loop on that
in my preallocation and assembly routines.


You can make an MPIAIJ matrix yourself of course. It should have the 
same division of rows as the DMDA division of dofs. Also, 
MatSetValuesStencil() will not work for a custom matrix.


  Thanks,

     Matt

Thanks very much,

Thibaut



--
What most experimenters take for granted before they begin their 
experiments is infinitely more interesting than any results to which 
their experiments lead.

-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/

Re: [petsc-users] Using PETSc in Cray systems

2019-02-21 Thread Matthew Knepley via petsc-users 
On Thu, Feb 21, 2019 at 10:46 AM Najib Alia via petsc-users <
petsc-users@mcs.anl.gov> wrote:

> Dear all,
>
> we are trying to compile our Finite Element code on a Cray system and
> have a problem with PETSc and available packages: "unable to find
> scotch64", the variable PETSC_SINGLE_LIBRARY is set to NOTFOUND, and the
> compiler tests fail.
>

First, we should see if PETSc works correctly.

  mkdir tmp
  cd tmp
  
  ex5: ex5.o
${CLINKER} -o ex5 ex5.o ${PETSC_LIB}

  include ${PETSC_DIR}/lib/petsc/conf/variables
  include ${PETSC_DIR}/lib/petsc/conf/rules
  https://bitbucket.org/petsc/petsc/src/master/src/snes/examples/tutorials/ex5.c
>
  
  make ex5
  ./ex5 -snes_monitor

If that runs, it looks like the errors are coming from your own build
system, which I don't think we
would understand.

  Thanks,

Matt

What we have done:
>
> 1) We loaded the corresponding modules:
>
> module load cray-tpsl
>
> module load cray-petsc-64
>
> 2) and set the PETSC_DIR and ARCH variables as given in "module display
> cray-petsc-64":
>
> ---
> /opt/cray/modulefiles/cray-petsc-64/3.7.4.0:
>
> setenv   PRGENV64 64
> conflict cray-petsc-complex
> conflict petsc-complex
> conflict cray-petsc
> conflict petsc
> conflict cray-hdf5
> setenv   CRAY_PETSC_DIR /opt/cray/petsc/3.7.4.0/real
> setenv   CRAY_PETSC_BASE_DIR /opt/cray/petsc/3.7.4.0/real
> setenv   CRAY_PETSC_VERSION 3.7.4.0
> setenv   CRAY_PETSC_PREFIX_DIR
> /opt/cray/petsc/3.7.4.0/real/CRAY64/8.3/sandybridge
> setenv   PETSC_DIR
> /opt/cray/petsc/3.7.4.0/real/CRAY64/8.3/sandybridge
> setenv   PETSC_ARCH
> prepend-path PE_PRODUCT_LIST CRAY_PETSC
> prepend-path CRAY_LD_LIBRARY_PATH
> /opt/cray/petsc/3.7.4.0/real/CRAY64/8.3/sandybridge/lib
> prepend-path MANPATH
> /opt/cray/petsc/3.7.4.0/real/man:/opt/cray/man/csmlversion
> setenv   PE_PETSC_MODULE_NAME cray-petsc-64/3.7.4.0
> setenv   PE_PETSC_VOLATILE_PRGENV CRAY CRAY64 GNU GNU64 INTEL
> INTEL64
> ...
> module-whatisPETSc - Portable, Extensible Toolkit for Scientific
> Computation
> ---
>
> 3) the library scotch is correctly installed (verified using "module
> display cray-tpsl").
>
> Our program uses the following file to find PETSc:
>
> https://github.com/jedbrown/cmake-modules/blob/master/FindPETSc.cmake
>
> Do you have suggestions ?
>
> How can we check the current configuration of PETSc and set the
> variables correctly? Is it normal that PETSC_ARCH is empty?
>
> Since we do not have much experience with Cray systems, we would like to
> avoid configure PETSc by our own.
>
> Thanks ahead for your suggestions,
>
> N. Alia
>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/

[petsc-users] About DMDA (and extracting its ordering)

2019-02-21 Thread Thibaut Appel via petsc-users 

Dear PETSc developers/users,

I’m solving linear PDEs on a regular grid with high-order finite 
differences, assembling an MPIAIJ matrix to solve linear systems or 
eigenvalue problems. I’ve been using vertex major, natural ordering for 
the parallelism with PetscSplitOwnership (yielding rectangular slices of 
the physical domain) and wanted to move to DMDA to have a more 
square-ish domain decomposition and minimize communication between 
processes.


However, my application is memory critical, and I have finely-tuned 
matrix preallocation routines for allocating memory “optimally”. It 
seems the memory of a DMDA matrix is allocated along the value of the 
stencil width of DMDACreate and the manual says about it


“These DMDA stencils have nothing directly to do with any finite 
difference stencils one might chose to use for a discretization”


And despite reading the manual pages there must be something I do not 
understand in the DM topology, what is that "stencil width" for then? I 
will not use ghost values for my FD-method, right?


I was then wondering if I could just create a MPIAIJ matrix, and with a 
PETSc routine get the global indices of the domain for each process: in 
other words, an equivalent of PetscSplitOwnership that gives me the DMDA 
unknown ordering. So I can feed and loop on that in my preallocation and 
assembly routines.


Thanks very much,

Thibaut

[petsc-users] Using PETSc in Cray systems

2019-02-21 Thread Najib Alia via petsc-users 
Dear all,

we are trying to compile our Finite Element code on a Cray system and
have a problem with PETSc and available packages: "unable to find
scotch64", the variable PETSC_SINGLE_LIBRARY is set to NOTFOUND, and the
compiler tests fail.

What we have done:

1) We loaded the corresponding modules:

module load cray-tpsl

module load cray-petsc-64

2) and set the PETSC_DIR and ARCH variables as given in "module display
cray-petsc-64":

---
/opt/cray/modulefiles/cray-petsc-64/3.7.4.0:

setenv   PRGENV64 64
conflict cray-petsc-complex
conflict petsc-complex
conflict cray-petsc
conflict petsc
conflict cray-hdf5
setenv   CRAY_PETSC_DIR /opt/cray/petsc/3.7.4.0/real
setenv   CRAY_PETSC_BASE_DIR /opt/cray/petsc/3.7.4.0/real
setenv   CRAY_PETSC_VERSION 3.7.4.0
setenv   CRAY_PETSC_PREFIX_DIR
/opt/cray/petsc/3.7.4.0/real/CRAY64/8.3/sandybridge
setenv   PETSC_DIR
/opt/cray/petsc/3.7.4.0/real/CRAY64/8.3/sandybridge
setenv   PETSC_ARCH
prepend-path PE_PRODUCT_LIST CRAY_PETSC
prepend-path CRAY_LD_LIBRARY_PATH
/opt/cray/petsc/3.7.4.0/real/CRAY64/8.3/sandybridge/lib
prepend-path MANPATH
/opt/cray/petsc/3.7.4.0/real/man:/opt/cray/man/csmlversion
setenv   PE_PETSC_MODULE_NAME cray-petsc-64/3.7.4.0
setenv   PE_PETSC_VOLATILE_PRGENV CRAY CRAY64 GNU GNU64 INTEL
INTEL64
...
module-whatis    PETSc - Portable, Extensible Toolkit for Scientific
Computation
---

3) the library scotch is correctly installed (verified using "module
display cray-tpsl").

Our program uses the following file to find PETSc:

https://github.com/jedbrown/cmake-modules/blob/master/FindPETSc.cmake

Do you have suggestions ?

How can we check the current configuration of PETSc and set the
variables correctly? Is it normal that PETSC_ARCH is empty?

Since we do not have much experience with Cray systems, we would like to
avoid configure PETSc by our own.

Thanks ahead for your suggestions,

N. Alia