Given the nonzero structure of the matrix I'd be surprised if either
GAMG/boomerAMG or ASM was particularly efficacious. I don't have any
recommendations for iterative methods.
Barry
> On Feb 11, 2019, at 8:41 AM, Abhyankar, Shrirang G via petsc-users
> wrote:
>
>
>
> From: Justin Chang
> Sent: Friday, February 8, 2019 4:57 PM
> To: Abhyankar, Shrirang G
> Cc: Mark Adams ; PETSc users list
> Subject: Re: [petsc-users] Preconditioning systems of equations with complex
> numbers
>
> So I used -mat_view draw -draw_pause -1 on my medium sized matrix and got
> this output:
>
>
>
> So it seems there are lots of off-diagonal terms, and that a decomposition of
> the problem via matload would give a terrible unbalanced problem.
>
> Given the initial A and b Mat/Vec, I experimented with MatPartioning and
> inserted the following lines into my code:
>
> Mat Apart;
> Vec bpart;
> MatPartitioning part;
> ISis,isrows;
> ierr = MatPartitioningCreate(PETSC_COMM_WORLD, &part);CHKERRQ(ierr);
> ierr = MatPartitioningSetAdjacency(part, A);CHKERRQ(ierr);
> ierr = MatPartitioningSetFromOptions(part);CHKERRQ(ierr);
> ierr = MatPartitioningApply(part, &is);CHKERRQ(ierr);
> ierr = ISBuildTwoSided(is,NULL,&isrows);CHKERRQ(ierr);
> ierr = MatCreateSubMatrix(A, isrows,isrows, MAT_INITIAL_MATRIX,
> &Apart);CHKERRQ(ierr);
> ierr = MatSetOptionsPrefix(Apart, "part_");CHKERRQ(ierr);
> ierr = MatSetFromOptions(Apart);CHKERRQ(ierr);
> ierr = VecGetSubVector(b,isrows,&bpart);CHKERRQ(ierr);
>
> /* Set Apart and bpart in the KSPSolve */
> ...
>
> And here are the mat_draw figures from 2 and 4 MPI processes respectively:
>
>
>
> Is this "right"? It just feels like I'm just duplicating the nnz structure
> among all the MPI processes. And it didn't really improve the performance of
> ASM.
>
> This looks kind of right. You have the big fat diagonal blocks represent
> separate meshed networks, and the sparse off-diagonal blocks that show very
> less connectivity between the networks. This is typical of power grid where
> the connectivity between different areas is sparse. However, the blocks being
> identical is puzzling. I would expect the blocks to be of different sizes.
> Was this network created by duplicating smaller networks?
> I was hoping ASM would be faster on the partitioned matrix rather than the
> original one. What options are you using for sub_pc? -sub_pc_type lu
> -sub_pc_factor_mat_ordering_type amd?
>
> Also, this nnz structure appears to depend on the distribution system we cook
> up in OpenDSS, for example the largest matrix we have thus far looks like
> this:
>
> Yes, the matrix structure depends on the network structure. This to me looks
> like a star-shaped network where you have a tightly interconnected network
> (bottom dense block) and smallish radial distribution networks emanating from
> the nodes. That’s just one guess. It could be some other type as well.
>
>
> Is there a better alternative to the partitioning I implemented above? Or did
> I do something catastrophically wrong
>
> Thanks,
> Justin
>
> Side node - Yes I tried KLU, but it needed several KSP iterations to obtain
> the solution. Here's the ksp monitor/view outputs:
>
> 0 KSP preconditioned resid norm 1.705434112839e+06 true resid norm
> 2.242813827253e+12 ||r(i)||/||b|| 1.e+00
> 1 KSP preconditioned resid norm 1.122965789284e+05 true resid norm
> 3.057749589444e+11 ||r(i)||/||b|| 1.363354172463e-01
> 2 KSP preconditioned resid norm 1.962518730076e+04 true resid norm
> 2.510054932552e+10 ||r(i)||/||b|| 1.119154386357e-02
> 3 KSP preconditioned resid norm 3.094963519133e+03 true resid norm
> 6.489763653495e+09 ||r(i)||/||b|| 2.893581078660e-03
> 4 KSP preconditioned resid norm 1.755871992454e+03 true resid norm
> 2.315676037474e+09 ||r(i)||/||b|| 1.032486963178e-03
> 5 KSP preconditioned resid norm 1.348939340771e+03 true resid norm
> 1.864929933344e+09 ||r(i)||/||b|| 8.315134812722e-04
> 6 KSP preconditioned resid norm 5.532203694243e+02 true resid norm
> 9.985525631209e+08 ||r(i)||/||b|| 4.452231170449e-04
> 7 KSP preconditioned resid norm 3.636087020506e+02 true resid norm
> 5.712899201028e+08 ||r(i)||/||b|| 2.547201703329e-04
> 8 KSP preconditioned resid norm 2.926812321412e+02 true resid norm
> 3.627282296417e+08 ||r(i)||/||b|| 1.617290856843e-04
> 9 KSP preconditioned resid norm 1.629184033135e+02 true resid norm
> 1.048838851435e+08 ||r(i)||/||b|| 4.676441881580e-05
> 10 KSP preconditioned resid norm 8.297821067807e+01 true resid norm
> 3.423640694920e+07 ||r(i)||/||b|| 1.526493484800e-05
> 11 KSP preconditioned resid norm 2.997246200648e+01 true resid norm
> 9.880250538293e+06 ||r(i)||/||b|| 4.405292324417e-06
> 12 KSP preconditioned resid norm 2.156940809471e+01 true resid norm
> 3.521518932572e+06 ||r(i)||/||b|| 1.570134306192e-06
> 13 KSP preconditioned resid norm 1.2118233084
