Sorry, I uses PCASMSetSortIndices for GASM. Now, I correct it and the output becomes the same as PCASM. See attached.
On May 21, 2012, at 8:59 PM, Hui Zhang wrote: > Dmitry, > > finally, I am able to make a simple program and reproduce the error. Since I > had moved to > petsc-dev (before I used petsc-3.2), the error is a little different. > > Download the program attached and run the run.sh. You will see the output > from PCASM. > There is a submatrix (submat[1] on rank 0) which has a row like > > row 1: (0, -1) (1, 0 + 0.4 i) (4, -1) (2, -1) > > Note that the column 2 is placed in the last and the column numbers are not > in the normal order. > Following that, we can also see the error message like > > [0]PETSC ERROR: New nonzero at (1,2) caused a malloc! > > You can also try with GASM, use > > mpirun -np 2 ./gasm_test -dd_type gasm -n 4 -px 1 -py 2 -sx 2 -sy 1 > > which partitions the domain to 1 * 2 and mapped to the two processors, and > for each processor > we further partition its local domain to 2 * 1 subdomains. > > Then, you will see that the row 1 of submat[1] on rank 1 becomes > > row 1: (0, -1) (1, 0 + 0.4 i) (3, -1) (6, -1) > > which is totally wrong because from the IS for this overlapping subdomain and > the big matrix A, > row 1 should have non-zeros like the output from PCASM. > > I guess the problem is due to that we set PCASMSetSortIndices to FALSE and > something goes wrong > in asm.c and gasm.c. > > Thanks! > Hui > > > <GASM_test.zip> > > > > > >> On Sun, May 20, 2012 at 5:06 AM, Hui Zhang <mike.hui.zhang at hotmail.com> >> wrote: >> I have the following MatView output: >> >> row 0: (0, -0.0351719 - 0.638814 i) (1, -0.517586 - 0.617823 i) (4, >> -0.517586 - 0.617823 i) (1, -1.03517) (5, -0.508793) >> >> Note that there are two values for the column 1, is this normal? >> >> The above matrix is obtained from PCASMGetSubKSP(pc,subksp) and >> KSPGetOperators(subksp[0], ...). >> >> This is odd. Can you explain a bit more what leads up to this, so we can try >> and reproduce the problem? >> Thanks. >> Dmitry. >> >> Thanks! >> >> >> >> On May 11, 2012, at 7:17 PM, Dmitry Karpeev wrote: >> >>> You can call PCSetUp(pc) on either ASM or GASM, and that will destroy and >>> recreate the matrices (including calling >>> your modification subroutine), but not the subdomains or the subdomain >>> solvers. >>> If you just want to modify the submatrices, you can call >>> PC(G)ASMGetSubmatrices() and modify the matrices it returns >>> (in the same order as the subdomains were set). That's a bit of a hack, >>> since you will essentially be modifying the PC's internal data structures. >>> As long as you are careful, you should be okay, since you already >>> effectively have the same type of access to the submatrices through the >>> Modify callback. >>> >>> Dmitry. >>> >>> On Fri, May 11, 2012 at 11:52 AM, Hui Zhang <mike.hui.zhang at hotmail.com> >>> wrote: >>> I just have a question about reuse of PCASM or PCGASM. >>> Suppose I have seted up the PCASM and related KSP and I solved one time. >>> Next for the same linear system (matrix and RHS), I just want PCASM modify >>> the submatrices (PCSetModifySubmatrices) in a different way, using the same >>> routine for modifying but with >>> different user context for the modifying routine. >>> >>> What can I do for this task? Currently, I destroy the KSP and re-construct >>> it. I guess >>> even for PCASM I can re-use it because the partition of subdomains remain >>> the same. >>> >>> Thanks! >>> >>> >>> On May 10, 2012, at 6:37 PM, Dmitry Karpeev wrote: >>> >>>> Hui, >>>> There've been several changes to PCGASM ahead of the new release. >>>> Let me go back and see if it affected the convergence problem. >>>> Dmitry. >>>> >>>> On Thu, May 10, 2012 at 4:16 AM, Hui Zhang <mike.hui.zhang at hotmail.com> >>>> wrote: >>>> Hi Dmitry, >>>> >>>> is there any news about PCGASM? >>>> >>>> thanks, >>>> Hui >>>> >>>> On Feb 20, 2012, at 6:38 PM, Dmitry Karpeev wrote: >>>> >>>>> Okay, thanks. >>>>> I'll take a look. >>>>> >>>>> Dmitry. >>>>> >>>>> On Mon, Feb 20, 2012 at 11:30 AM, Hui Zhang <mike.hui.zhang at >>>>> hotmail.com> wrote: >>>>> For reference, my results are attached. >>>>> >>>>> asm1.txt for asm with 1 process, >>>>> asm2.txt for asm with 2 processes, >>>>> gasm1.txt for gasm with 1 process, (with the iteration numbers different >>>>> from others) >>>>> gasm2.txt for gasm with 2 processes >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> thank you, >>>>> Hui >>>>> >>>>> On Feb 20, 2012, at 3:06 PM, Dmitry Karpeev wrote: >>>>> >>>>>> >>>>>> >>>>>> On Mon, Feb 20, 2012 at 12:59 AM, Hui Zhang <mike.hui.zhang at >>>>>> hotmail.com> wrote: >>>>>> >>>>>> On Feb 20, 2012, at 12:41 AM, Dmitry Karpeev wrote: >>>>>> >>>>>>> >>>>>>> >>>>>>> On Sun, Feb 19, 2012 at 3:08 PM, Hui Zhang <mike.hui.zhang at >>>>>>> hotmail.com> wrote: >>>>>>> I have a new problem: the results from ASM and GASM are different and >>>>>>> it seems >>>>>>> GASM has something wrong with SetModifySubMatrices. Numerical tests are >>>>>>> with >>>>>>> each subdomain supported only by one subdomain. There are no problems >>>>>>> when >>>>>>> I did not modify submatrices. But when I modify submatrices, there are >>>>>>> problems >>>>>>> with GASM but no problems with ASM. >>>>>>> >>>>>>> For example, I use two subdomains. In the first case each subdomain is >>>>>>> supported by >>>>>>> one processor and there seems no problem with GASM. But when I use run >>>>>>> my program >>>>>>> with only one proc. so that it supports both of the two subdomains, the >>>>>>> iteration >>>>>>> number is different from the first case and is much larger. On the >>>>>>> other hand >>>>>>> ASM has no such problem. >>>>>>> >>>>>>> Are the solutions the same? >>>>>>> What problem are you solving? >>>>>> >>>>>> Yes, the solutions are the same. That's why ASM gives the same results >>>>>> with one or >>>>>> two processors. But GASM did not. >>>>>> Sorry, I wasn't clear: ASM and GASM produced different solutions in the >>>>>> case of two domains per processor? >>>>>> I'm solving the Helmholtz equation. Maybe >>>>>> I can prepare a simpler example to show this difference. >>>>>> That would be helpful. >>>>>> Thanks. >>>>>> >>>>>> Dmitry. >>>>>> >>>>>>> >>>>>>> Dmitry. >>>>>>> >>>>>>> >>>>>>> On Feb 15, 2012, at 6:46 PM, Dmitry Karpeev wrote: >>>>>>> >>>>>>>> You should be able to. >>>>>>>> This behavior is the same as in PCASM, >>>>>>>> except in GASM the matrices live on subcommunicators. >>>>>>>> I am in transit right now, but I can take a closer look in Friday. >>>>>>>> >>>>>>>> Dmitry >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Feb 15, 2012, at 8:07, Hui Zhang <mike.hui.zhang at hotmail.com> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> On Feb 15, 2012, at 11:19 AM, Hui Zhang wrote: >>>>>>>>> >>>>>>>>>> Hi Dmitry, >>>>>>>>>> >>>>>>>>>> thanks a lot! Currently, I'm not using ISColoring. Just comes >>>>>>>>>> another question >>>>>>>>>> on PCGASMSetModifySubMatrices(). The user provided function has the >>>>>>>>>> prototype >>>>>>>>>> >>>>>>>>>> func (PC pc,PetscInt nsub,IS *row,IS *col,Mat *submat,void *ctx); >>>>>>>>>> >>>>>>>>>> I think the coloumns from the parameter 'col' are always the same as >>>>>>>>>> the rows >>>>>>>>>> from the parameter 'row'. Because PCGASMSetLocalSubdomains() only >>>>>>>>>> accepts >>>>>>>>>> index sets but not rows and columns. Has I misunderstood something? >>>>>>>>> >>>>>>>>> As I tested, the row and col are always the same. >>>>>>>>> >>>>>>>>> I have a new question. Am I allowed to SetLocalToGlobalMapping() for >>>>>>>>> the submat's >>>>>>>>> in the above func()? >>>>>>>>> >>>>>>>>> thanks, >>>>>>>>> Hui >>>>>>>>> >>>>>>>>>> >>>>>>>>>> thanks, >>>>>>>>>> Hui >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Feb 11, 2012, at 3:36 PM, Dmitry Karpeev wrote: >>>>>>>>>> >>>>>>>>>>> Yes, that's right. >>>>>>>>>>> There is no good way to help the user assemble the subdomains at >>>>>>>>>>> the moment beyond the 2D stuff. >>>>>>>>>>> It is expected that they are generated from mesh subdomains. >>>>>>>>>>> Each IS does carry the subdomains subcomm. >>>>>>>>>>> >>>>>>>>>>> There is ISColoringToList() that is supposed to convert a >>>>>>>>>>> "coloring" of indices to an array of ISs, >>>>>>>>>>> each having the indices with the same color and the subcomm that >>>>>>>>>>> supports that color. It is >>>>>>>>>>> largely untested, though. You could try using it and give us >>>>>>>>>>> feedback on any problems you encounter. >>>>>>>>>>> >>>>>>>>>>> Dmitry. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Sat, Feb 11, 2012 at 6:06 AM, Hui Zhang <mike.hui.zhang at >>>>>>>>>>> hotmail.com> wrote: >>>>>>>>>>> About PCGASMSetLocalSubdomains(), in the case of one subdomain >>>>>>>>>>> supported by >>>>>>>>>>> multiple processors, shall I always create the arguments 'is[s]' >>>>>>>>>>> and 'is_local[s]' >>>>>>>>>>> in a subcommunicator consisting of processors supporting the >>>>>>>>>>> subdomain 's'? >>>>>>>>>>> >>>>>>>>>>> The source code of PCGASMCreateSubdomains2D() seemingly does so. >>>>>>>>>>> >>>>>>>>>>> Thanks, >>>>>>>>>>> Hui >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>> >>>>> >>>>> >>>> >>>> >>> >>> >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120521/b78f83b4/attachment-0002.html> -------------- next part -------------- A non-text attachment was scrubbed... Name: GASM_test.zip Type: application/zip Size: 7094 bytes Desc: not available URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120521/b78f83b4/attachment-0001.zip> -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120521/b78f83b4/attachment-0003.html>
