To repeat: You seem to be saying that OpenFOAM solves the problem in 10 seconds and PETSc solves it in 14 seconds. Is that correct?
On Thu, Nov 8, 2018 at 3:42 AM Edoardo alinovi via petsc-users < petsc-users@mcs.anl.gov> wrote: > Hello Mark, > > Yes, there are 5 KSP calls within a time-step (3 for the solution of > momentum equation + 2 for the solution of pressure), this is the classical > non iterative PISO by Issa ( the exact sequence of operations is : solve > momentum implicitly, solve pressure-correction, momentum explicitly, > pressure correction). The pressure correction equation ,which is something > similar to a Poisson equation for incompressible flows, is the one that > determines the overall performance in my code such as in the others. > Usually, when the pressure is being solved for the second time, the > solution is faster since there is a better input guess and, as in my case, > the preconditioner is not recomputed again. > > Have you got some advices for the multigrid configuration in this > scenario, which are not the default one, in order to increase performances? > > I do not know if this may impact drastically the performance, but I am > running on a E4 workstation with 16 Intel's Xeon processors (2.3GH/12MB > cache) and 128GB of RAM . > > Thank you very much for your helpful comments, > > > Edoardo > ------ > > Edoardo Alinovi, Ph.D. > > DICCA, Scuola Politecnica > Universita' di Genova > 1, via Montallegro > 16145 Genova, Italy > > email: edoardo.alin...@dicca.unige.it > Tel: +39 010 353 2540 > > > > > Il giorno mer 7 nov 2018 alle ore 17:59 Mark Adams <mfad...@lbl.gov> ha > scritto: > >> please respond to petsc-users. >> >> You are doing 5 solves here in 14 seconds. You seem to be saying that the >> two pressure solves are taking all of this time. I don't know why the two >> solves are different. >> >> You seem to be saying that OpenFOAM solves the problem in 10 seconds and >> PETSc solves it in 14 seconds. Is that correct? Hypre seems to be running >> fine. >> >> >> >> On Wed, Nov 7, 2018 at 11:24 AM Edoardo alinovi < >> edoardo.alin...@gmail.com> wrote: >> >>> Thanks a lot Mark for your kind replay. The solver is mine and I use >>> PETSc for the solution of momentum and pressure. The first is solved very >>> fast by a standard bcgs + bjacobi, but the pressure is the source of all >>> evils and, unfortunately, I am pretty sure that almost all the time within >>> the time-step is needed by KSP to solve the pressure (see log attached). I >>> have verified this also putting a couple of mpi_wtime around the kspsolve >>> call. The pressure is solved 2 times (1 prediction + 1 correction), the >>> prediction takes around 11s , the correction around 4s (here I am avoiding >>> to recompute the preconditioner), all the rest of the code (flux assembling >>> + mometum solution + others) around 1s. Openfoam does the same procedure >>> with the same tolerance in 10s using its gamg version (50 it to converge). >>> The number of iteration required to solve the pressure with hypre are 12. >>> Gamg performs similarly to hypre in terms of speed, but with 50 iterations >>> to converge. Am I missing something in the setup in your opinion? >>> >>> thanks a lot, >>> >>> Edo >>> >>> ------ >>> >>> Edoardo Alinovi, Ph.D. >>> >>> DICCA, Scuola Politecnica >>> Universita' di Genova >>> 1, via Montallegro >>> 16145 Genova, Italy >>> >>> email: edoardo.alin...@dicca.unige.it >>> Tel: +39 010 353 2540 >>> >>> >>> >>> >>> Il giorno mer 7 nov 2018 alle ore 16:50 Mark Adams <mfad...@lbl.gov> ha >>> scritto: >>> >>>> You can try -pc_type gamg, but hypre is a pretty good solver for the >>>> Laplacian. If hypre is just a little faster than LU on a 3D problem (that >>>> takes 10 seconds to solve) then AMG is not doing well. I would expect that >>>> AMG is taking a lot of iterations (eg, >> 10). You can check that with >>>> -ksp_monitor. >>>> >>>> The PISO algorithm is a multistage algorithm with a pressure correction >>>> in it. It also has a solve for the velocity, from what I can tell. Are you >>>> building PISO yourself and using PETSc just for the pressure correction? >>>> Are you sure the time is spent in this solver? You can use -log_view to see >>>> performance numbers and look for KSPSolve to see how much time is spent in >>>> the PETSc solver. >>>> >>>> Mark >>>> >>>> >>>> On Wed, Nov 7, 2018 at 10:26 AM Zhang, Hong via petsc-maint < >>>> petsc-ma...@mcs.anl.gov> wrote: >>>> >>>>> Edoardo: >>>>> Forwarding your request to petsc-maint where you can get fast and >>>>> expert advise. I do not have suggestion for your application, but someone >>>>> in our team likely will make suggestion. >>>>> Hong >>>>> >>>>> Hello Hong, >>>>>> >>>>>> Well, using -sub_pc_type lu it super slow. I am desperately triying >>>>>> to enhance performaces of my code (CFD, finite volume, PISO alghoritm), >>>>>> in >>>>>> particular I have a strong bottleneck in the solution of pressure >>>>>> correction equation which takes almost the 90% of computational time. >>>>>> Using >>>>>> multigrid as preconditoner (hypre with default options) is slighlty >>>>>> better, but comparing the results against the multigrid used in openFOAM, >>>>>> my code is losing 10s/iteration which a huge amount of time. Now, since >>>>>> that all the time is employed by KSPSolve, I feel a bit powerless. Do >>>>>> you >>>>>> have any helpful advice? >>>>>> >>>>>> Thank you very much! >>>>>> ------ >>>>>> >>>>>> Edoardo Alinovi, Ph.D. >>>>>> >>>>>> DICCA, Scuola Politecnica >>>>>> Universita' di Genova >>>>>> 1, via Montallegro >>>>>> 16145 Genova, Italy >>>>>> >>>>>> email: edoardo.alin...@dicca.unige.it >>>>>> Tel: +39 010 353 2540 >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> Il giorno mar 6 nov 2018 alle ore 17:15 Zhang, Hong < >>>>>> hzh...@mcs.anl.gov> ha scritto: >>>>>> >>>>>>> Edoardo: >>>>>>> Interesting. I thought it would not affect performance much. What >>>>>>> happens if you use -sub_pc_type lu'? >>>>>>> Hong >>>>>>> >>>>>>> Dear Hong and Matt, >>>>>>>> >>>>>>>> thank you for your kind replay. I have just tested your suggestions >>>>>>>> and applied " -sub_pc_type ilu -sub_pc_factor_mat_ordering_type nd/rcm" >>>>>>>> and, in both cases, I have found a deterioration of performances >>>>>>>> with respect to doing nothing (thus just putting default PCBJACOBI). >>>>>>>> Is it >>>>>>>> normal? However, I guess this is very problem dependent. >>>>>>>> ------ >>>>>>>> >>>>>>>> Edoardo Alinovi, Ph.D. >>>>>>>> >>>>>>>> DICCA, Scuola Politecnica >>>>>>>> Universita' di Genova >>>>>>>> 1, via Montallegro >>>>>>>> 16145 Genova, Italy >>>>>>>> >>>>>>>> email: edoardo.alin...@dicca.unige.it >>>>>>>> Tel: +39 010 353 2540 >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> Il giorno mar 6 nov 2018 alle ore 16:04 Zhang, Hong < >>>>>>>> hzh...@mcs.anl.gov> ha scritto: >>>>>>>> >>>>>>>>> Edoardo: >>>>>>>>> You can test runtime option '-sub_pc_factor_mat_ordering_type' and >>>>>>>>> use '-log_view' to get performance on different orderings, >>>>>>>>> e.g.,petsc/src/ksp/ksp/examples/tutorials/ex2.c: >>>>>>>>> mpiexec -n 2 ./ex2 -ksp_view -sub_pc_type ilu >>>>>>>>> -sub_pc_factor_mat_ordering_type nd >>>>>>>>> >>>>>>>>> I do not think the ordering inside block for ilu would affect >>>>>>>>> performance much. Let us know what you will get. >>>>>>>>> Hong >>>>>>>>> >>>>>>>>> Dear users, >>>>>>>>>> >>>>>>>>>> I have a question about the correct use of the option >>>>>>>>>> "PCFactorSetMatOrderingType" in PETSc. >>>>>>>>>> >>>>>>>>>> I am solving a problem with 2.5M of unknowns distributed along 16 >>>>>>>>>> processors and I am using the block jacobi preconditioner and MPIAIJ >>>>>>>>>> matrix format. I cannot figure out if the above option can be useful >>>>>>>>>> or not >>>>>>>>>> in decreasing the computational time. Any suggestion or tips? >>>>>>>>>> >>>>>>>>>> Thank you very much for the kind help >>>>>>>>>> >>>>>>>>>> ------ >>>>>>>>>> >>>>>>>>>> Edoardo Alinovi, Ph.D. >>>>>>>>>> >>>>>>>>>> DICCA, Scuola Politecnica >>>>>>>>>> Universita' di Genova >>>>>>>>>> 1, via Montallegro >>>>>>>>>> 16145 Genova, Italy >>>>>>>>>> >>>>>>>>>> email: edoardo.alin...@dicca.unige.it >>>>>>>>>> Tel: +39 010 353 2540 >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>