Hi Matt, very interesting project you are working on. We haven't gone deep on how we would do this in GPUs and are starting to look at options. We will explore if it is possible to batch work needed for several cells within a thread group on the gpu.
We use a single Cartesian mesh per MPI process (usually with 40^3 to 50^3 cells). Something I implemented to avoid the MPI process over-subscription of GPU with PETSc solvers was to cluster several MPI Processes per GPU on resource sets. Then, the processes in the set would pass matrix (at setup) and RHS to a single process (set master) which communicates with the GPU. The GPU solution is then brought back to the set master which distributes it to the MPI processes in the set as needed. So, only a set of processes as large as the number of GPUs in the calculation (with their own MPI communicator) call the PETSc matrix and vector building, and solve routines. The neat thing is that all MPI communications are local to the node. This idea is not new, it was developed by the researchers at GWU that interfaced PETSc to AMGx back when there were no native GPU solvers in PETSc, HYPRE and other libs (~2016). Best, Marcos ________________________________ From: Matthew Knepley <knep...@gmail.com> Sent: Monday, October 16, 2023 4:31 PM To: Vanella, Marcos (Fed) <marcos.vane...@nist.gov> Cc: petsc-users@mcs.anl.gov <petsc-users@mcs.anl.gov>; Paul, Chandan (IntlAssoc) <chandan.p...@nist.gov> Subject: Re: [petsc-users] Using Sundials from PETSc On Mon, Oct 16, 2023 at 4:08 PM Vanella, Marcos (Fed) <marcos.vane...@nist.gov<mailto:marcos.vane...@nist.gov>> wrote: Hi Mathew, we have code that time splits the combustion step from the chemical species transport, so on each computational cell for each fluid flow time step, once transport is done we have the mixture chemical composition as initial condition. We are looking into doing finite rate chemistry with skeletal combustion models (20+ equations) in each cell for each fluid time step. Sundials provides the CVODE solver for the time integration of these, and would be interesting to see if we can make use of GPU acceleration. From their User Guide for Version 6.6.0 there are several GPU implementations for building RHS and using linear, nonlinear and stiff ODE solvers. We are doing a similar thing in CHREST (https://www.buffalo.edu/chrest.html). Since we normally use hundreds of species and thousands of reactions for the reduced mechanism, we are using TChem2 to build and solve the system in each cell. Since these systems are so small, you are likely to need some way of batching them within a warp. Do you have an idea for this already? Thanks, Matt Thank you Satish for the comment. Might be better at this point to first get an idea on what the implementation in our code using Sundials directly would look like. Then, we can see if it is possible and makes sense to access it through PETSc. We have things working in CPU making use of and older version of CVODE. BTW after some changes in our code we are starting running larger cases using GPU accelerated iterative solvers from PETSc, so we have PETSc interfaced already. Thanks! ________________________________ From: Matthew Knepley <knep...@gmail.com<mailto:knep...@gmail.com>> Sent: Monday, October 16, 2023 3:03 PM To: Vanella, Marcos (Fed) <marcos.vane...@nist.gov<mailto:marcos.vane...@nist.gov>> Cc: petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov> <petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov>>; Paul, Chandan (IntlAssoc) <chandan.p...@nist.gov<mailto:chandan.p...@nist.gov>> Subject: Re: [petsc-users] Using Sundials from PETSc On Mon, Oct 16, 2023 at 2:29 PM Vanella, Marcos (Fed) via petsc-users <petsc-users@mcs.anl.gov<mailto:petsc-users@mcs.anl.gov>> wrote: Hi, we were wondering if it would be possible to call the latest version of Sundials from PETSc? The short answer is, no. We are at v2.5 and they are at v6.5. There were no dates on the version history page, so I do not know how out of date we are. There have not been any requests for update until now. We would be happy to get an MR for the updates if you want to try it. We are interested in doing chemistry using GPUs and already have interfaces to PETSc from our code. How does the GPU interest interact with the SUNDIALS version? Thanks, Matt Thanks, Marcos -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/<http://www.cse.buffalo.edu/~knepley/> -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/<http://www.cse.buffalo.edu/~knepley/>
