On Fri, Jan 21, 2022 at 11:14 AM Jed Brown <j...@jedbrown.org> wrote:
> "Paul T. Bauman" <ptbau...@gmail.com> writes: > > > On Fri, Jan 21, 2022 at 8:52 AM Paul T. Bauman <ptbau...@gmail.com> > wrote: > >> Yes. The way HYPRE's memory model is setup is that ALL GPU allocations > are > >> "native" (i.e. [cuda,hip]Malloc) or, if unified memory is enabled, then > ALL > >> GPU allocations are unified memory (i.e. [cuda,hip]MallocManaged). > >> Regarding HIP, there is an HMM implementation of hipMallocManaged > planned, > >> but is it not yet delivered AFAIK (and it will *not* support gfx906, > e.g. > >> RVII, FYI), so, today, under the covers, hipMallocManaged is calling > >> hipHostMalloc. So, today, all your unified memory allocations in HYPRE > on > >> HIP are doing CPU-pinned memory accesses. And performance is just truly > >> terrible (as you might expect). > > Thanks for this important bit of information. > > And it sounds like when we add support to hand off Kokkos matrices and > vectors (our current support for matrices on ROCm devices uses Kokkos) or > add direct support for hipSparse, we'll avoid touching host memory in > assembly-to-solve with hypre. > It does not look like anyone has made Hypre work with HIP. Stafano added a runex19_hypre_hip target 4 months ago and hypre.py has some HIP things. I have a user that would like to try this, no hurry but, can I get an idea of a plan for this? Thanks, Mark