Note that options like -mat_type aijcusparse -vec_type cuda only work if the program is set up to allow runtime swapping of matrix and vector types. If you have a call to MatCreateMPIAIJ() or other specific types then then these options do nothing but because Mark had you use -options_left the program will tell you at the end that it did not use the option so you will know.
> On Jun 30, 2023, at 9:30 AM, Mark Adams <mfad...@lbl.gov> wrote: > > PetscCall(PetscInitialize(&argc, &argv, NULL, help)); gives us the args and > you run: > > a.out -mat_type aijcusparse -vec_type cuda -log_view -options_left > > Mark > > On Fri, Jun 30, 2023 at 6:16 AM Matthew Knepley <knep...@gmail.com > <mailto:knep...@gmail.com>> wrote: >> On Fri, Jun 30, 2023 at 1:13 AM Ng, Cho-Kuen via petsc-users >> <petsc-users@mcs.anl.gov <mailto:petsc-users@mcs.anl.gov>> wrote: >>> Mark, >>> >>> The application code reads in parameters from an input file, where we can >>> put the PETSc runtime options. Then we pass the options to >>> PetscInitialize(...). Does that sounds right? >> >> PETSc will read command line argument automatically in PetscInitialize() >> unless you shut it off. >> >> Thanks, >> >> Matt >> >>> Cho >>> From: Ng, Cho-Kuen <c...@slac.stanford.edu <mailto:c...@slac.stanford.edu>> >>> Sent: Thursday, June 29, 2023 8:32 PM >>> To: Mark Adams <mfad...@lbl.gov <mailto:mfad...@lbl.gov>> >>> Cc: petsc-users@mcs.anl.gov <mailto:petsc-users@mcs.anl.gov> >>> <petsc-users@mcs.anl.gov <mailto:petsc-users@mcs.anl.gov>> >>> Subject: Re: [petsc-users] Using PETSc GPU backend >>> >>> Mark, >>> >>> Thanks for the information. How do I put the runtime options for the >>> executable, say, a.out, which does not have the provision to append >>> arguments? Do I need to change the C++ main to read in the options? >>> >>> Cho >>> From: Mark Adams <mfad...@lbl.gov <mailto:mfad...@lbl.gov>> >>> Sent: Thursday, June 29, 2023 5:55 PM >>> To: Ng, Cho-Kuen <c...@slac.stanford.edu <mailto:c...@slac.stanford.edu>> >>> Cc: petsc-users@mcs.anl.gov <mailto:petsc-users@mcs.anl.gov> >>> <petsc-users@mcs.anl.gov <mailto:petsc-users@mcs.anl.gov>> >>> Subject: Re: [petsc-users] Using PETSc GPU backend >>> >>> Run with options: -mat_type aijcusparse -vec_type cuda -log_view >>> -options_left >>> >>> The last column of the performance data (from -log_view) will be the >>> percent flops on the GPU. Check that that is > 0. >>> >>> The end of the output will list the options that were used and options that >>> were _not_ used (if any). Check that there are no options left. >>> >>> Mark >>> >>> On Thu, Jun 29, 2023 at 7:50 PM Ng, Cho-Kuen via petsc-users >>> <petsc-users@mcs.anl.gov <mailto:petsc-users@mcs.anl.gov>> wrote: >>> I installed PETSc on Perlmutter using "spack install petsc+cuda+zoltan" and >>> used it by "spack load petsc/fwge6pf". Then I compiled the application code >>> (purely CPU code) linking to the petsc package, hoping that I can get >>> performance improvement using the petsc GPU backend. However, the timing >>> was the same using the same number of MPI tasks with and without GPU >>> accelerators. Have I missed something in the process, for example, setting >>> up PETSc options at runtime to use the GPU backend? >>> >>> Thanks, >>> Cho >> >> >> -- >> What most experimenters take for granted before they begin their experiments >> is infinitely more interesting than any results to which their experiments >> lead. >> -- Norbert Wiener >> >> https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
