Thanks for having a look at it. I look forward to giving it a try when it implemented. And like bjacobi, support for mpibaij would be very beneficial to me.
Thank you Gaetan On Mon, May 20, 2013 at 12:08 PM, Hong Zhang <[email protected]> wrote: > Gaetan: > >> >> On a related note, I tried using the ASM version of the same approach; >> that is -pc_type asm -pc_asm_blocks 4 with the remainder of the options >> the same. This gives a message that the number of blocks is less than the >> number of processors (sorry I don't have the exact message anymore). I get >> this error with both mpiaij and mpibaij types. >> > > Thanks for reporting it. > The '_Multiproc' version of asm is not implemented yet, although it is > logically same as bjacobi_Multiproc. > I added to our 'to-do' list > https://bitbucket.org/petsc/petsc/issue/42/implement-pcsetup_asm_multiproc > >> >> Has this approach been implemented/do you think there would be any >> benefit from the approach? >> > It would be beneficial for applications that give better convergence and > performance with asm than bjacobi. > It is necessary to provide such support in petsc library. > We'll let you know when it is implemented. > > Hong > >> >> On Mon, May 20, 2013 at 10:34 AM, Hong Zhang <[email protected]> wrote: >> >>> Gaetan : >>> >>>> >>>> It runs if the mattype is mpiaij instead of mpibaij. I gather this is >>>> not implemented for the blocked matrix types? >>> >>> It is not tested for mpibaij format yet. I'll check it. >>> The paper uses mpiaij format. >>> >>> Hong >>> >>>> >>>> Gaetan >>>> >>>> On Mon, May 20, 2013 at 9:26 AM, Gaetan Kenway <[email protected]>wrote: >>>> >>>>> Hi again >>>>> >>>>> I installed petsc3.4.0 and I am still getting the following error when >>>>> running with the following options (on 64 procs) >>>>> >>>>> # Matrix Options >>>>> -matload_block_size 5 -mat_type mpibaij >>>>> >>>>> # KSP solver options >>>>> -ksp_type fgmres -ksp_max_it 1000 -ksp_gmres_restart 200 -ksp_monitor >>>>> -ksp_view -ksp_pc_side right -ksp_rtol 1e-6 >>>>> >>>>> # Nested GMRES Options >>>>> -pc_type bjacobi -pc_bjacobi_blocks 4 -sub_ksp_type gmres >>>>> -sub_ksp_max_it 5 -sub_pc_type bjacobi -sub_sub_pc_type ilu >>>>> -sub_sub_pc_factor_mat_ordering_type rcm -sub_sub_pc_factor_levels 1 >>>>> >>>>> Any thoughts? >>>>> >>>>> Thank you, >>>>> >>>>> Gaetan >>>>> >>>>> [44]PETSC ERROR: >>>>> ------------------------------------------------------------------------ >>>>> [44]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, >>>>> probably memory access out of range >>>>> [44]PETSC ERROR: Try option -start_in_debugger or >>>>> -on_error_attach_debugger >>>>> [44]PETSC ERROR: or see >>>>> http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind[44]PETSCERROR: >>>>> or try >>>>> http://valgrind.org on GNU/linux and Apple Mac OS X to find memory >>>>> corruption errors >>>>> [44]PETSC ERROR: likely location of problem given in stack below >>>>> [44]PETSC ERROR: --------------------- Stack Frames >>>>> ------------------------------------ >>>>> [44]PETSC ERROR: Note: The EXACT line numbers in the stack are not >>>>> available, >>>>> [44]PETSC ERROR: INSTEAD the line number of the start of the >>>>> function >>>>> [44]PETSC ERROR: is given. >>>>> [44]PETSC ERROR: [44] PCSetUp_BJacobi_Multiproc line 1197 >>>>> /home/j/jmartins/kenway/packages/petsc-3.4.0/src/ksp/pc/impls/bjacobi/bjacobi.c >>>>> [44]PETSC ERROR: [44] PCSetUp_BJacobi line 24 >>>>> /home/j/jmartins/kenway/packages/petsc-3.4.0/src/ksp/pc/impls/bjacobi/bjacobi.c >>>>> [44]PETSC ERROR: [44] PCSetUp line 868 >>>>> /home/j/jmartins/kenway/packages/petsc-3.4.0/src/ksp/pc/interface/precon.c >>>>> [44]PETSC ERROR: [44] KSPSetUp line 192 >>>>> /home/j/jmartins/kenway/packages/petsc-3.4.0/src/ksp/ksp/interface/itfunc.c >>>>> [44]PETSC ERROR: [44] KSPSolve line 356 >>>>> /home/j/jmartins/kenway/packages/petsc-3.4.0/src/ksp/ksp/interface/itfunc.c >>>>> [43]PETSC ERROR: >>>>> ------------------------------------------------------------------------ >>>>> >>>>> >>>>> On Sun, May 19, 2013 at 11:15 PM, Barry Smith <[email protected]>wrote: >>>>> >>>>>> >>>>>> You should be using PETSc version 3.4 which was recently released >>>>>> and is what the paper is based on. >>>>>> >>>>>> Barry >>>>>> >>>>>> On May 19, 2013, at 10:11 PM, Gaetan Kenway <[email protected]> >>>>>> wrote: >>>>>> >>>>>> > Hi Everyone >>>>>> > >>>>>> > I am trying to replicate the type of preconditioner described in >>>>>> "Hierarchical and Nested Krylov Methods for Extreme-Scale Computing". >>>>>> > >>>>>> > I have used the following options: (I'm using fortran so the >>>>>> following is my petsc_options file) >>>>>> > >>>>>> > # Matrix Options >>>>>> > -matload_block_size 5 >>>>>> > -mat_type mpibaij >>>>>> > >>>>>> > # KSP solver options >>>>>> > -ksp_type gmres >>>>>> > -ksp_max_it 1000 >>>>>> > -ksp_gmres_restart 200 >>>>>> > -ksp_monitor >>>>>> > -ksp_view >>>>>> > -ksp_pc_side right >>>>>> > -ksp_rtol 1e-6 >>>>>> > >>>>>> > # Nested GMRES Options >>>>>> > -pc_type bjacobi >>>>>> > -pc_bjacobi_blocks 4 >>>>>> > -sub_ksp_type gmres >>>>>> > -sub_ksp_max_it 5 >>>>>> > -sub_pc_type bjacobi >>>>>> > -sub_sub_pc_type ilu >>>>>> > -sub_sub_pc_factor_mat_ordering_type rcm >>>>>> > -sub_sub_pc_factor_levels 1 >>>>>> > >>>>>> > The test is run on 64 processors and the total number of block >>>>>> jacobi blocks is 4 (less than nproc). The error I get is: >>>>>> > >>>>>> > [6]PETSC ERROR: Note: The EXACT line numbers in the stack are not >>>>>> available, >>>>>> > [6]PETSC ERROR: INSTEAD the line number of the start of the >>>>>> function >>>>>> > [6]PETSC ERROR: is given. >>>>>> > [6]PETSC ERROR: [6] PCSetUp_BJacobi_Multiproc line 1269 >>>>>> /home/j/jmartins/kenway/packages/petsc-3.3-p5/src/ksp/pc/impls/bjacobi/bjacobi.c >>>>>> > [6]PETSC ERROR: [6] PCSetUp_BJacobi line 24 >>>>>> /home/j/jmartins/kenway/packages/petsc-3.3-p5/src/ksp/pc/impls/bjacobi/bjacobi.c >>>>>> > [6]PETSC ERROR: [6] PCSetUp line 810 >>>>>> /home/j/jmartins/kenway/packages/petsc-3.3-p5/src/ksp/pc/interface/precon.c >>>>>> > [6]PETSC ERROR: [6] KSPSetUp line 182 >>>>>> /home/j/jmartins/kenway/packages/petsc-3.3-p5/src/ksp/ksp/interface/itfunc.c >>>>>> > [6]PETSC ERROR: [6] KSPSolve line 351 >>>>>> /home/j/jmartins/kenway/packages/petsc-3.3-p5/src/ksp/ksp/interface/itfunc.c >>>>>> > [6]PETSC ERROR: --------------------- Error Message >>>>>> ------------------------------------ >>>>>> > [6]PETSC ERROR: Signal received! >>>>>> > [6]PETSC ERROR: >>>>>> ------------------------------------------------------------------------ >>>>>> > [6]PETSC ERROR: Petsc Release Version 3.3.0, Patch 5, Sat Dec 1 >>>>>> 15:10:41 CST 2012 >>>>>> > [6]PETSC ERROR: See docs/changes/index.html for recent updates. >>>>>> > [6]PETSC ERROR: See docs/faq.html for hints about trouble shooting. >>>>>> > [6]PETSC ERROR: >>>>>> ------------------------------------------------------------------------ >>>>>> > [6]PETSC ERROR: >>>>>> ------------------------------------------------------------------------ >>>>>> > [6]PETSC ERROR: ./main on a intel-rea named gpc-f109n001 by kenway >>>>>> Sun May 19 23:01:52 2013 >>>>>> > [6]PETSC ERROR: Libraries linked from >>>>>> /home/j/jmartins/kenway/packages/petsc-3.3-p5/intel-real-debug/lib >>>>>> > [6]PETSC ERROR: Configure run at Sun Jan 20 15:52:20 2013 >>>>>> > [6]PETSC ERROR: Configure options --with-shared-libraries >>>>>> --download-superlu_dist=yes --download-parmetis=yes --download-metis=yes >>>>>> --with-fortran-interfaces=1 --with-debugging=yes --with-scalar-type=real >>>>>> -with-petsc-arch=intel-real-debug --with-blas-lapack-dir= --with-pic >>>>>> > [6]PETSC ERROR: >>>>>> ------------------------------------------------------------------------ >>>>>> > >>>>>> > If the number of blocks is greater than or equal to the number of >>>>>> processors it runs fine. I'm using version 3.3-p5. >>>>>> > >>>>>> > The options as listed in the paper are: >>>>>> > -flow_ksp_type fgmres -flow_ksp_pc_side right -flow_pc_type bjacobi >>>>>> -flow_pc_bjacobi_blocks ngp >>>>>> > -flow_sub_ksp_type gmres -flow_sub_ksp_max_it 6 -flow_sub_pc_type >>>>>> bjacobi >>>>>> > -flow_sub_sub_pc_type ilu >>>>>> > >>>>>> > Any suggestions would be greatly appreciated. >>>>>> > >>>>>> > Thank you, >>>>>> > >>>>>> > Gaetan Kenway >>>>>> > >>>>>> > >>>>>> >>>>>> >>>>> >>>> >>> >> >
