Are you using https://urldefense.us/v3/__https://develop.openfoam.com/modules/external-solver__;!!G_uCfscf7eWS!YgXDdUKnfJdPEbKAQzR8TYlQzciWTcMmU_6jrKexdBB6UajcOHUvJhfaBjBtMN2cX6EUFDrcF4LcFItaSLxbAb0$
If so you might be able to add it like petsc { options { log_view; ksp_type cg; pc_type bjacobi; sub_pc_type icc; } as discussed on the webpage for setting PETSc solver options > On Jun 14, 2024, at 3:44 PM, Sircar, Arpan <sirc...@ornl.gov> wrote: > > Hi Junchao and Barry, > > I tried adding -log_view to my OpenFOAM command since that is what I am > running the entire package through. However, it does not recognize that as a > valid option. I am not sure where to put that tag in this setup. > I am however using a Kokkos profiling tool, the output of which (for the GPU > run) is attached with this email. Do let me know if this is useful or if you > have ideas of where to put the -log_view tag. > > Junchao - Great nice running into you here. > > Thanks, > Arpan > From: Junchao Zhang <junchao.zh...@gmail.com <mailto:junchao.zh...@gmail.com>> > Sent: Friday, June 14, 2024 3:16 PM > To: Sircar, Arpan <sirc...@ornl.gov <mailto:sirc...@ornl.gov>> > Cc: Barry Smith <bsm...@petsc.dev <mailto:bsm...@petsc.dev>>; > petsc-users@mcs.anl.gov <mailto:petsc-users@mcs.anl.gov> > <petsc-users@mcs.anl.gov <mailto:petsc-users@mcs.anl.gov>>; Gottiparthi, > Kalyan <gottipart...@ornl.gov <mailto:gottipart...@ornl.gov>> > Subject: Re: [petsc-users] [EXTERNAL] Re: Running PETSc with a Kokkos backend > on OLCF Frontier > > Arpan, > Did you add -log_view ? > > --Junchao Zhang > > > On Fri, Jun 14, 2024 at 2:00 PM Sircar, Arpan via petsc-users > <petsc-users@mcs.anl.gov <mailto:petsc-users@mcs.anl.gov>> wrote: > This Message Is From an External Sender > This message came from outside your organization. > > Hi Barry, > > Thanks for your prompt response. These are run with with the same PETSc > solvers but the one on GPUs (log_pKok) uses mataijkokkos while the other one > does not. > > Please let me know if you need any other information. > > Thanks, > Arpan > From: Barry Smith <bsm...@petsc.dev <mailto:bsm...@petsc.dev>> > Sent: Friday, June 14, 2024 1:47 PM > To: Sircar, Arpan <sirc...@ornl.gov <mailto:sirc...@ornl.gov>> > Cc: petsc-users@mcs.anl.gov <mailto:petsc-users@mcs.anl.gov> > <petsc-users@mcs.anl.gov <mailto:petsc-users@mcs.anl.gov>>; Gottiparthi, > Kalyan <gottipart...@ornl.gov <mailto:gottipart...@ornl.gov>> > Subject: [EXTERNAL] Re: [petsc-users] Running PETSc with a Kokkos backend on > OLCF Frontier > > > Please run both the CPU solvers and GPU solvers cases with -log_view and > send the two outputs. > > Barry > > >> On Jun 14, 2024, at 1:35 PM, Sircar, Arpan via petsc-users >> <petsc-users@mcs.anl.gov <mailto:petsc-users@mcs.anl.gov>> wrote: >> >> This Message Is From an External Sender >> This message came from outside your organization. >> Hi, >> >> We have been working with OpenFOAM (an open-source CFD software) which can >> transfer its matrices to PETSc to use its linear solvers. This has been >> tested and is working well on OCLF's Frontier machine. Next we are trying to >> use the Kokkos backend to run it on Frontier GPUs. While the >> OpenFOAM+PETSc+Kokkos environment built correctly on Frontier using the >> modules sourced (attached file bash_petsc4foam_gpu) and configuring PETSc >> correctly (attached file config_gpu), the GPU solve seems to take more time >> than the CPU solve. >> >> The PETSc run-time options we are using are attached to this email (file >> fvSolution_petsc_pKok_Uof). Could you please take a look and let us know if >> this combination of options is fine? In this approach we are trying to solve >> the pressure equation only on the GPUs. >> >> Thanks, >> Arpan >> >> Arpan Sircar >> R&D Associate Staff >> Thermal Hydraulics Group >> Nuclear Energy and Fuel Cycle Division >> Oak Ridge National Laboratory >> >> <fvSolution_petsc_pKok_Uof><bash_petsc4foam_gpu><config_gpu> > > <log.kok>