So your operator is a "dense" operator in that its matrix representation would be essentially dense. In that case, you can use specialized routines to do this efficiently. You can use VecScatterCreateToAll() and then VecScatterBegin/End to do the communication.
Barry > On Jul 6, 2022, at 1:39 PM, Tu, Jiannan <jiannan...@uml.edu> wrote: > > Jed, thank you very much for the reply. > > I'm not sure GlobaltoLocal will work. Say the solution vector is of length > 10. Two processes then each process can see 5 elements of the vector. For A x > = b (10 equations), each process performs matrix-vector product for five > equations. And for each equation i , sum_j A_ij * x_j = b_i requires the > process has access to all 10 unknows x_i (i=0,1,...,9). > > Can VecScatter and PetscSF make the entire vector accessible to each process? > I am reading the manual and trying to understand how VecScatter and PetscSF > work. > > Jiannan > From: Jed Brown <j...@jedbrown.org <mailto:j...@jedbrown.org>> > Sent: Wednesday, July 6, 2022 10:09 AM > To: Tu, Jiannan <jiannan...@uml.edu <mailto:jiannan...@uml.edu>>; Barry Smith > <bsm...@petsc.dev <mailto:bsm...@petsc.dev>> > Cc: petsc-users@mcs.anl.gov <mailto:petsc-users@mcs.anl.gov> > <petsc-users@mcs.anl.gov <mailto:petsc-users@mcs.anl.gov>> > Subject: Re: [petsc-users] Using matrix-free with KSP > > You'll usually have a GlobalToLocal operation for each rank to get the halo > data it needs, then either a LocalToGlobal to collect the result (e.g., > finite element methods) or the local compute will write directly into the > owned portion of the global vector. If you're doing the communication "raw", > then you may find VecScatter or PetscSF useful to perform the necessary > communication. > > "Tu, Jiannan" <jiannan...@uml.edu <mailto:jiannan...@uml.edu>> writes: > > > Hi Barry, > > > > Following your instructions I implemented the matrix-free method to solve a > > large linear equation system resulted from a surface integration equation. > > The KSP solver works fine for single process, but it is problematic with > > multiple processes. The problem is that each process only can access its > > own part of solution vector so that each process only conducts part of > > matrix-vector multiplication. MPI can be used to assemble these partial > > matrix-vector multiplication together. Does Petsc provide any ways to > > implement multi-process matrix-free method? > > > > Thanks, > > Jiannan > > ________________________________ > > From: Barry Smith <bsm...@petsc.dev <mailto:bsm...@petsc.dev>> > > Sent: Tuesday, May 24, 2022 2:12 PM > > To: Tu, Jiannan <jiannan...@uml.edu <mailto:jiannan...@uml.edu>> > > Cc: petsc-users@mcs.anl.gov <mailto:petsc-users@mcs.anl.gov> > > <petsc-users@mcs.anl.gov <mailto:petsc-users@mcs.anl.gov>> > > Subject: Re: [petsc-users] Using matrix-free with KSP > > > > This e-mail originated from outside the UMass Lowell network. > > ________________________________ > > > > You can use MatCreateMFFD > > https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanualpages%2FMat%2FMatCreateMFFD%2F&data=05%7C01%7CJiannan_Tu%40uml.edu%7Cab0859d8d154471590bc08da5f5918d6%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637927133525745809%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=A7wqegTfh94No5BpDiWLK3VxOuR44U2wlWHVm2k7l60%3D&reserved=0<https://urldefense.com/v3/__https://petsc.org/main/docs/manualpages/Mat/MatCreateMFFD/__;!!PVKG_VDCxu5g!stxq2NWGVjUihvHBQC9Rlk0aHVoowcy0PTQYMcQxQ4DqOEC05KSw6TmstSXKckiUgelHzy4ue-d10-zlXkw$> > > > > <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanualpages%2FMat%2FMatCreateMFFD%2F&data=05%7C01%7CJiannan_Tu%40uml.edu%7Cab0859d8d154471590bc08da5f5918d6%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637927133525745809%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=A7wqegTfh94No5BpDiWLK3VxOuR44U2wlWHVm2k7l60%3D&reserved=0%3Chttps://urldefense.com/v3/__https://petsc.org/main/docs/manualpages/Mat/MatCreateMFFD/__;!!PVKG_VDCxu5g!stxq2NWGVjUihvHBQC9Rlk0aHVoowcy0PTQYMcQxQ4DqOEC05KSw6TmstSXKckiUgelHzy4ue-d10-zlXkw$%3E> > > MatMFFDSetFunction MatSetFromOptions MatMFFDSetBase and provide the matrix > > to KSP. Note you will need to use -pc_type none or provide your own custom > > preconditioner > > withhttps://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanualpages%2FPC%2FPCSHELL%2F&data=05%7C01%7CJiannan_Tu%40uml.edu%7Cab0859d8d154471590bc08da5f5918d6%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637927133525745809%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=Gb%2F9uFG3jp%2FbfWaSh2cSEQLItHS9CuLwarzN0KcNiJY%3D&reserved=0<https://urldefense.com/v3/__https://petsc.org/main/docs/manualpages/PC/PCSHELL/__;!!PVKG_VDCxu5g!stxq2NWGVjUihvHBQC9Rlk0aHVoowcy0PTQYMcQxQ4DqOEC05KSw6TmstSXKckiUgelHzy4ue-d1cA4fKGw$> > > > > <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org%2Fmain%2Fdocs%2Fmanualpages%2FPC%2FPCSHELL%2F&data=05%7C01%7CJiannan_Tu%40uml.edu%7Cab0859d8d154471590bc08da5f5918d6%7C4c25b8a617f746f983f054734ab81fb1%7C0%7C0%7C637927133525745809%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=Gb%2F9uFG3jp%2FbfWaSh2cSEQLItHS9CuLwarzN0KcNiJY%3D&reserved=0%3Chttps://urldefense.com/v3/__https://petsc.org/main/docs/manualpages/PC/PCSHELL/__;!!PVKG_VDCxu5g!stxq2NWGVjUihvHBQC9Rlk0aHVoowcy0PTQYMcQxQ4DqOEC05KSw6TmstSXKckiUgelHzy4ue-d1cA4fKGw$%3E> > > > > > > > > On May 24, 2022, at 1:21 PM, Tu, Jiannan <jiannan...@uml.edu > > <mailto:jiannan...@uml.edu><mailto:jiannan...@uml.edu > > <mailto:jiannan...@uml.edu>>> wrote: > > > > I want to use a matrix-free matrix to solve a large linear equation system > > because the matrix is too large to be stored. Petsc user manual describes > > matrix-free method for SNES with examples. The matrix-free matrices section > > explains how to set up such a matrix, but I can't find any example of > > matrix-free method with KSP. I am wondering how to apply the method to KSP > > iterative solver in parallel. > > > > I greatly appreciate your help for me to understand this topic. > > > > Jiannan Tu
