http://www.mcs.anl.gov/petsc/documentation/faq.html#efficient-assembly
On Jul 12, 2013, at 8:46 AM, "Salko, Robert K." <salk...@ornl.gov> wrote: > I'm attempting to implement PETSc for the parallel solution of a large sparse > matrix in a computer program I work with. The program is written in Fortran. > I currently have been able to get the program working using PETSc and giving > correct results, but the issue is its performance. It is much slower than > solving the matrix in serial. I seem to have narrowed the major PETSc time > sink down to actually constructing the matrix (getting the matrix terms into > the PETSc matrix). Specifically, it is hanging for a very long time to do > this step: > > do values_counter = 1, total_values > call > MatSetValues(A,1,coeff_row(values_counter),1,coeff_col(values_counter),coeff(values_counter),INSERT_VALUES,petsc_err) > end do > > Building the right-hand-side vector for this matrix is much faster: > > call VecSetValues(y,nrows,index_of_values,rhs_values,INSERT_VALUES,petsc_err) > > The difference, as I see it, is that I'm sending the matrix coefficients to > the PETSc matrix one element at a time whereas with the RHS vector, I'm > sending all the values in one shot. There are 7 times more elements in the > matrix than the vector, so I get that it will take longer. So I did some > timing studies using MPI_Wtime and putting the matrix values into the PETSc > matrix is taking 4,800 times longer than putting the values into the RHS > vector. > > Then there is the actual assembly of the matrix that you have to do using > MatAssemblyBegin and MatAssemblyEnd. That takes even longer than assigning > values to the matrix. More than twice as long. Actually solving the matrix > and then distributing the solution back to the domains in the program takes > about as much time as the RHS vector assignment step, so that is nearly 5,000 > times faster than putting data into PETSc. Surely there must be something > wrong that I'm doing with the assignment of data into PETSc. > > So my question is, is there a more efficient way to build the matrix? I > already have my coefficients stored in AIJ form in the program. I understand > that PETSc uses AIJ. Can't I somehow just send the three AIJ vectors to PETSc > in one shot? > > Thanks, > Bob
