Dear Barry, Thanks for your response. Now I am doing simple debugging for my customized mat_mult_op of my shell jacobian matrix. As far as I understand, because the input and output of shell jacobian are all global vectors, I need to do the global to local mapping of my input vector (dF) by myself. Before starting debugging the mapping, I first try to verify the size match of input and output. Because the input and output of a mat_mult_op of my shell matrix should have the same size. So I tried just equating them in my customized mat_mult_op. Basically like this subroutine GK_op(Jac,dF_global,output_global,err_PETSc) DM :: dm_local ! Yi: later for is,ie Vec :: dF_global Vec :: output_global PetscErrorCode :: err_PETSc real(pREAL), pointer,dimension(:,:,:,:) :: dF_scal, output_scal call SNESGetDM(SNES_mech,dm_local,err_PETSc) CHKERRQ(err_PETSc) output_global = dF_global end subroutine GK_op When I run with mpirun -np 3, it gives me similar error like previous, “Preconditioner number of local rows 27 does not equal input vector size 81”, (I changed my problem size so the numbers are different). Maybe simple equating input and output is not valid (due to ownership of different portion of a global vector). Then it may give me different error message. In fact my global dF has size 9*3*3*3, when running on 3 processors, the local dF has size 9*3*3*1 (I split my domain in z direction). The error message seems to suggest I am using a local dF rather than a global dF. And the output and input vector sizes seems to be different. Do I miss something here? Best regards, Yi From: Barry Smith <bsm...@petsc.dev> Sent: Sunday, March 10, 2024 6:45 PM To: Yi Hu <y...@mpie.de> Cc: Mark Adams <mfad...@lbl.gov>; petsc-users <petsc-users@mcs.anl.gov> Subject: Re: [petsc-users] snes matrix-free jacobian fails with preconditioner shape mismatch
On Mar 10, 2024, at 10:16 AM, Yi Hu <y...@mpie.de> wrote: This Message Is From an External Sender This message came from outside your organization. Dear Mark, Thanks for your reply. I see this mismatch. In fact my global DoF is 324. It seems like I always get the local size = global Dof / np^2, np is my processor number. By the way, I used DMDASNESsetFunctionLocal() to set my form function. Is it eligible to mix DMDASNESsetFunctionLocal() and a native SNESSetJacobian()? Yes Best, Yi On 3/10/24 13:55, Mark Adams wrote: It looks like your input vector is the global vector, size 162, and the local matrix size is 81.Mark[1]PETSC ERROR: Preconditioner number of local rows 81 does not equal input vector size 162 On Sun, Mar 10, 2024 at 7:21 AM Yi Hu <y...@mpie.de> wrote: This Message Is From an External Sender This message came from outside your organization. Dear petsc team, I implemented a matrix-free jacobian, and it can run sequentially. But running parallel I got the pc error like this (running with mpirun -np 2, only error from rank1 is presented here) [1]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------[1]PETSC ERROR: Nonconforming object sizes[1]PETSC ERROR: Preconditioner number of local rows 81 does not equal input vector size 162[1]PETSC ERROR: See https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G_uCfscf7eWS!ahmistzr4wD3TJ0OvI0JWxB9aVSIbP78Jcs2X_6KMb4LdoR8drLB_DkHvaguhrca22RgFer0PlyUrtdfCA$ for trouble shooting.[1]PETSC ERROR: Petsc Release Version 3.17.3, Jun 29, 2022[1]PETSC ERROR: /home/yi/workspace/DAMASK_yi/bin/DAMASK_grid on a arch-linux-c-opt named carbon-x1 by yi Sun Mar 10 12:01:46 2024[1]PETSC ERROR: Configure options --download-fftw --download-hdf5 --with-hdf5-fortran-bindings --download-fblaslapack --download-chaco --download-hypre --download-metis --download-mumps --download-parmetis --download-scalapack --download-suitesparse --download-superlu --download-superlu_dist --download-triangle --download-zlib --download-cmake --with-cxx-dialect=C++11 --with-c2html=0 --with-debugging=0 --with-ssl=0 --with-x=0 COPTFLAGS=-O3 CXXOPTFLAGS=-O3 FOPTFLAGS=-O3[1]PETSC ERROR: #1 PCApply() at /home/yi/App/petsc-3.17.3/src/ksp/pc/interface/precon.c:424[1]PETSC ERROR: #2 KSP_PCApply() at /home/yi/App/petsc-3.17.3/include/petsc/private/kspimpl.h:376[1]PETSC ERROR: #3 KSPInitialResidual() at /home/yi/App/petsc-3.17.3/src/ksp/ksp/interface/itres.c:64[1]PETSC ERROR: #4 KSPSolve_GMRES() at /home/yi/App/petsc-3.17.3/src/ksp/ksp/impls/gmres/gmres.c:242[1]PETSC ERROR: #5 KSPSolve_Private() at /home/yi/App/petsc-3.17.3/src/ksp/ksp/interface/itfunc.c:902[1]PETSC ERROR: #6 KSPSolve() at /home/yi/App/petsc-3.17.3/src/ksp/ksp/interface/itfunc.c:1078[1]PETSC ERROR: #7 SNESSolve_NEWTONLS() at /home/yi/App/petsc-3.17.3/src/snes/impls/ls/ls.c:222[1]PETSC ERROR: #8 SNESSolve() at /home/yi/App/petsc-3.17.3/src/snes/interface/snes.c:4756[1]PETSC ERROR: #9 User provided function() at User file:0 However, from snes matrix-free documentation (https://urldefense.us/v3/__https://petsc.org/release/manual/snes/*matrix-free-methods__;Iw!!G_uCfscf7eWS!ahmistzr4wD3TJ0OvI0JWxB9aVSIbP78Jcs2X_6KMb4LdoR8drLB_DkHvaguhrca22RgFer0Ply6ZbywOw$), it is said matrix-free is used with pcnone. So I assume it would not apply preconditioner, but it did use preconditioning probably the same as my matrix-free shell matrix. Here is how i initialize my shell matrix and the corresponding customized multiplication. call MatCreateShell(PETSC_COMM_WORLD,PETSC_DECIDE,PETSC_DECIDE,&int(9*product(cells(1:2))*cells3,pPETSCINT),&int(9*product(cells(1:2))*cells3,pPETSCINT),& F_PETSc,Jac_PETSc,err_PETSc) call MatShellSetOperation(Jac_PETSc,MATOP_MULT,GK_op,err_PETSc) call SNESSetDM(SNES_mech,DM_mech,err_PETSc) call SNESSetJacobian(SNES_mech,Jac_PETSc,Jac_PETSc,PETSC_NULL_FUNCTION,0,err_PETSc) call SNESGetKSP(SNES_mech,ksp,err_PETSc) call PCSetType(pc,PCNONE,err_PETSc) And my GK_op is like subroutine GK_op(Jac,dF_global,output_local,err_PETSc) DM :: dm_local Vec :: dF_global, dF_local, output_local Mat :: Jac PetscErrorCode :: err_PETSc real(pREAL), pointer,dimension(:,:,:,:) :: dF_scal, output_scal real(pREAL), dimension(3,3,cells(1),cells(2),cells3) :: & dF real(pREAL), dimension(3,3,cells(1),cells(2),cells3) :: & output call SNESGetDM(SNES_mech,dm_local,err_PETSc) call DMGetLocalVector(dm_local,dF_local,err_PETSc) call DMGlobalToLocalBegin(dm_local,dF_global,INSERT_VALUES,dF_local,err_PETSc) call DMGlobalToLocalEnd(dm_local,dF_global,INSERT_VALUES,dF_local,err_PETSc) call DMDAVecGetArrayReadF90(dm_local,dF_local,dF_scal,err_PETSc) dF = reshape(dF_scal, [3,3,cells(1),cells(2),cells3]) ....... call DMDAVecRestoreArrayF90(dm_local,output_local,output_scal,err_PETSc) CHKERRQ(err_PETSc) call DMDAVecRestoreArrayF90(dm_local,dF_local,dF_scal,err_PETSc) CHKERRQ(err_PETSc) end subroutine GK_op I checked my cells3, it corresponds to my local size, and it seems the local size of dF_local is ok. I am a bit lost here to find the reason for the preconditioner bug. Could you help me on this? Thanks. Best regards, Yi -------------------------------------------------Stay up to date and follow us on LinkedIn, Twitter and YouTube. Max-Planck-Institut für Eisenforschung GmbHMax-Planck-Straße 1D-40237 Düsseldorf Handelsregister B 2533 Amtsgericht Düsseldorf GeschäftsführungProf. Dr. Gerhard DehmProf. Dr. Jörg NeugebauerProf. Dr. Dierk RaabeDr. Kai de Weldige Ust.-Id.-Nr.: DE 11 93 58 514 Steuernummer: 105 5891 1000 Please consider that invitations and e-mails of our institute are only valid if they end with …@mpie.de. If you are not sure of the validity please contact r...@mpie.de Bitte beachten Sie, dass Einladungen zu Veranstaltungen und E-Mailsaus unserem Haus nur mit der Endung …@mpie.de gültig sind. In Zweifelsfällen wenden Sie sich bitte an r...@mpie.de------------------------------------------------- ------------------------------------------------- Stay up to date and follow us on LinkedIn, Twitter and YouTube. Max-Planck-Institut für Eisenforschung GmbH Max-Planck-Straße 1 D-40237 Düsseldorf Handelsregister B 2533 Amtsgericht Düsseldorf Geschäftsführung Prof. Dr. Gerhard Dehm Prof. Dr. Jörg Neugebauer Prof. Dr. Dierk Raabe Dr. Kai de Weldige Ust.-Id.-Nr.: DE 11 93 58 514 Steuernummer: 105 5891 1000 Please consider that invitations and e-mails of our institute are only valid if they end with …@mpie.de. If you are not sure of the validity please contact r...@mpie.de Bitte beachten Sie, dass Einladungen zu Veranstaltungen und E-Mails aus unserem Haus nur mit der Endung …@mpie.de gültig sind. In Zweifelsfällen wenden Sie sich bitte an r...@mpie.de ------------------------------------------------- ------------------------------------------------- Stay up to date and follow us on LinkedIn, Twitter and YouTube. Max-Planck-Institut für Eisenforschung GmbH Max-Planck-Straße 1 D-40237 Düsseldorf Handelsregister B 2533 Amtsgericht Düsseldorf Geschäftsführung Prof. Dr. Gerhard Dehm Prof. Dr. Jörg Neugebauer Prof. Dr. Dierk Raabe Dr. Kai de Weldige Ust.-Id.-Nr.: DE 11 93 58 514 Steuernummer: 105 5891 1000 Please consider that invitations and e-mails of our institute are only valid if they end with …@mpie.de. If you are not sure of the validity please contact r...@mpie.de Bitte beachten Sie, dass Einladungen zu Veranstaltungen und E-Mails aus unserem Haus nur mit der Endung …@mpie.de gültig sind. In Zweifelsfällen wenden Sie sich bitte an r...@mpie.de -------------------------------------------------
