Re: [petsc-users] hypre / hip usage

2022-01-24 Thread Mark Adams
On Sat, Jan 22, 2022 at 11:31 AM Stefano Zampini wrote: > Mark > > the two options are only there to test the code in CI, and are not needed > in general > >'--download-hypre-configure-arguments=--enable-unified-memory', > This is only here to test the unified memory code path > > '--with

Re: [petsc-users] hypre / hip usage

2022-01-24 Thread Paul T. Bauman
On Mon, Jan 24, 2022 at 9:53 AM Jed Brown wrote: > "Paul T. Bauman" writes: > > > 1. `rocgdb` will be in your PATH when the `rocm` module is loaded. This > is > > gdb, but with some extra AMDGPU goodies. AFAIK, you cannot, yet, do > > stepping through a kernel in the source (only the ISA), but y

Re: [petsc-users] hypre / hip usage

2022-01-24 Thread Mark Adams
OK, you meant in gdb. rocgdb seems to be hung here. Do you see a problem? Thanks, + srun -n1 -N1 --ntasks-per-gpu=1 --gpu-bind=closest rocgdb --args ../ex13 -dm_plex_box_faces 2,2,2 -petscpartitioner_simple_process_grid 2,2,2 -dm_plex_box_upper 1,1,1 -petscpartitioner_simple_node_grid 1,1,1 -dm_re

Re: [petsc-users] hypre / hip usage

2022-01-24 Thread Jed Brown
"Paul T. Bauman" writes: > 1. `rocgdb` will be in your PATH when the `rocm` module is loaded. This is > gdb, but with some extra AMDGPU goodies. AFAIK, you cannot, yet, do > stepping through a kernel in the source (only the ISA), but you can query > device variables in host code, print their valu

Re: [petsc-users] hypre / hip usage

2022-01-24 Thread Paul T. Bauman
On Mon, Jan 24, 2022 at 9:31 AM Mark Adams wrote: > Thanks Paul, > > How do I get a stack trace? I have been relying on PETSc's > which piggybacks on timers so it is not getting too deep here. > I'm not sure what the "PETSc way" is, but I just run the executable through `rocgdb` as one would do

Re: [petsc-users] hypre / hip usage

2022-01-24 Thread Mark Adams
Sorry for the dumb question , but I added print statements in my hypre code and want to rebuild hypre to get these changes. How does one do that? 'make all' in the hypre directory does not do it. Thanks, Mark On Mon, Jan 24, 2022 at 10:31 AM Mark Adams wrote: > Thanks Paul, > > How do I get a s

Re: [petsc-users] hypre / hip usage

2022-01-24 Thread Mark Adams
Thanks Paul, How do I get a stack trace? I have been relying on PETSc's which piggybacks on timers so it is not getting too deep here. On Mon, Jan 24, 2022 at 10:16 AM Paul T. Bauman wrote: > On Mon, Jan 24, 2022 at 8:53 AM Matthew Knepley wrote: > >> On Mon, Jan 24, 2022 at 9:24 AM Mark Adams

Re: [petsc-users] hypre / hip usage

2022-01-24 Thread Paul T. Bauman
On Mon, Jan 24, 2022 at 8:53 AM Matthew Knepley wrote: > On Mon, Jan 24, 2022 at 9:24 AM Mark Adams wrote: > >> What is the fastest way to rebuild hypre? reconfiguring did not work and >> is slow. >> >> I am printf debugging to find this HSA_STATUS_ERROR_MEMORY_FAULT (no >> debuggers other than

Re: [petsc-users] hypre / hip usage

2022-01-24 Thread Matthew Knepley
On Mon, Jan 24, 2022 at 9:24 AM Mark Adams wrote: > What is the fastest way to rebuild hypre? reconfiguring did not work and > is slow. > > I am printf debugging to find this HSA_STATUS_ERROR_MEMORY_FAULT (no > debuggers other than valgrind on Crusher??!?!) and I get to a hypre call: > > > Petsc

Re: [petsc-users] hypre / hip usage

2022-01-24 Thread Mark Adams
What is the fastest way to rebuild hypre? reconfiguring did not work and is slow. I am printf debugging to find this HSA_STATUS_ERROR_MEMORY_FAULT (no debuggers other than valgrind on Crusher??!?!) and I get to a hypre call: PetscStackCallStandard(HYPRE_IJMatrixAddToValues,(hA->ij,1,&hnc,(HYPRE

Re: [petsc-users] hypre / hip usage

2022-01-23 Thread Mark Adams
Stefano and Matt, This segv looks like a Plexism. + srun -n1 -N1 --ntasks-per-gpu=1 --gpu-bind=closest ../ex13 -dm_plex_box_faces 2,2,2 -petscpartitioner_simple_process_grid 1,1,1 -dm_plex_box_upper 1,1,1 -petscpartitioner_simple_node_grid 1,1,1 -dm_refine 2 -dm_view -malloc_debug -log_trace -pc_t

Re: [petsc-users] hypre / hip usage

2022-01-23 Thread Mark Adams
Thanks, '-mat_type hypre' was failing for me. I could not find a test that used it and I was not sure it was considered functional. I will look at it again and collect a bug report if needed. On Sat, Jan 22, 2022 at 11:31 AM Stefano Zampini wrote: > Mark > > the two options are only there to tes

Re: [petsc-users] hypre / hip usage

2022-01-22 Thread Stefano Zampini
Mark the two options are only there to test the code in CI, and are not needed in general '--download-hypre-configure-arguments=--enable-unified-memory', This is only here to test the unified memory code path '--with-hypre-gpuarch=gfx90a', This is not needed if rocminfo is in PATH Our in

Re: [petsc-users] hypre / hip usage

2022-01-21 Thread Mark Adams
On Fri, Jan 21, 2022 at 11:14 AM Jed Brown wrote: > "Paul T. Bauman" writes: > > > On Fri, Jan 21, 2022 at 8:52 AM Paul T. Bauman > wrote: > >> Yes. The way HYPRE's memory model is setup is that ALL GPU allocations > are > >> "native" (i.e. [cuda,hip]Malloc) or, if unified memory is enabled, th

Re: [petsc-users] hypre / hip usage

2022-01-21 Thread Jed Brown
"Paul T. Bauman" writes: > On Fri, Jan 21, 2022 at 8:52 AM Paul T. Bauman wrote: >> Yes. The way HYPRE's memory model is setup is that ALL GPU allocations are >> "native" (i.e. [cuda,hip]Malloc) or, if unified memory is enabled, then ALL >> GPU allocations are unified memory (i.e. [cuda,hip]Mall

Re: [petsc-users] hypre / hip usage

2022-01-21 Thread Jed Brown
Mark Adams writes: >> >> >> >> > Is there a way to tell from log_view data that hypre is running on the >> GPU? >> >> Is it clear from data transfer within PCApply? >> >> > Well, this does not look right. '-mat_type hypre' fails. I guess we have to > get that working or could/should it work with

Re: [petsc-users] hypre / hip usage

2022-01-21 Thread Paul T. Bauman
Dammit, didn't reply-all. Sorry. On Fri, Jan 21, 2022 at 8:52 AM Paul T. Bauman wrote: > > > On Fri, Jan 21, 2022 at 8:37 AM Jed Brown wrote: > >> "Paul T. Bauman" writes: >> >> > On Fri, Jan 21, 2022 at 8:19 AM Jed Brown wrote: >> > >> >> Mark Adams writes: >> >> >> >> > Two questions about

Re: [petsc-users] hypre / hip usage

2022-01-21 Thread Mark Adams
> > > > > Is there a way to tell from log_view data that hypre is running on the > GPU? > > Is it clear from data transfer within PCApply? > > Well, this does not look right. '-mat_type hypre' fails. I guess we have to get that working or could/should it work with -mat_type aijkokkos ? --- Event S

Re: [petsc-users] hypre / hip usage

2022-01-21 Thread Jed Brown
"Paul T. Bauman" writes: > On Fri, Jan 21, 2022 at 8:19 AM Jed Brown wrote: > >> Mark Adams writes: >> >> > Two questions about hypre on HIP: >> > >> > * I am doing this now. Is this correct? >> > >> > '--download-hypre', >> > '--download-hypre-configure-arguments=--enable-unified-memor

Re: [petsc-users] hypre / hip usage

2022-01-21 Thread Paul T. Bauman
On Fri, Jan 21, 2022 at 8:19 AM Jed Brown wrote: > Mark Adams writes: > > > Two questions about hypre on HIP: > > > > * I am doing this now. Is this correct? > > > > '--download-hypre', > > '--download-hypre-configure-arguments=--enable-unified-memory', > Apologies for interjecting, but

Re: [petsc-users] hypre / hip usage

2022-01-21 Thread Jed Brown
Mark Adams writes: > Two questions about hypre on HIP: > > * I am doing this now. Is this correct? > > '--download-hypre', > '--download-hypre-configure-arguments=--enable-unified-memory', > '--with-hypre-gpuarch=gfx90a', I's recommended to use --with-hip-arch=gfx90a, which forwards

[petsc-users] hypre / hip usage

2022-01-21 Thread Mark Adams
Two questions about hypre on HIP: * I am doing this now. Is this correct? '--download-hypre', '--download-hypre-configure-arguments=--enable-unified-memory', '--with-hypre-gpuarch=gfx90a', * -mat_type hypre fails, so I am not using a -mat_type now. Just -vec_type hip. Hypre does seem