Dear all,
I know this is a bit of a vague (and drawn out) question, but I hope
that someone will be able to give me advise regardless of this.
I am trying to convert my problem to solve the linear system on a SMP
(shared memory) machine. So its not distributed computing as such, but
rather a single large multicore machine. So I already have as many
functions as possible making use of TBB, which sees all of the cores and
uses them to the full extent possible. However, since I need to use a
SSOR preconditioner, the bottleneck is the linear solver for me since
this the .vmult() operation in the deal.II implementation is not
multithreaded.
I also cannot simply convert my entire problem to using MPI due to some
complex interactions within the program (I'm sure its not impossible,
just impractical at this time). So, what I'm trying to do is to simply
distribute the linear problem across the cores using the classes in the
Trilinos MPI wrappers (similar to step 17 I suppose). I suppose you can
say its using a mater-slave approach, where the main program is run on
the master process and the slave process wait around to be told to run
certain functions.
I have most of the logistics of dealing with the problem sorted, but get
stuck right at the important part. Initially, I make the grid and use
metis to partition it (since this allows access to all cells from any
process - it doesn't look like using p4est allows for this) on EACH
process. I then send n-1 processes to sleep and run the main part of the
program on process 0. Assembly of the system matrix and rhs etc. is
therefore done on the 0th (master) process. After assembly, the other
n-1 cores are waiting for the master process to tell them to go to the
solver function. I then copy the system matrix and rhs etc, to the other
processes (they all appear to have full and complete copies of these
objects), instruct all processes to run the solver function in which I
set up the Trilinos MPI matrix and vectors and solver, and then solve
the problem and pass the solution back to the 0th process.
I'm coming unstuck at correctly setting up the Trilinos matrices and
vectors. I've tried numerous permutations of the Trilinos::SparseMatrix
and Trilinos::MPI::Vector setup and initialisations using deal.II
matrices and either end up with some sort of segmentation fault, a
Trilinos error "-1" or, for my best attempt, a hung solver. I can get
the local dofs using the
"DoFTools::dof_indices_with_subdomain_association" function (it appears
that ghost cells might be included in the filled IndexSet), but as I
said I haven't been able to get the right combination of
mpi_matrix.reinit() and mpi_vector.reinit() calls to get this thing to work.
I really have tried to ingest the information in steps 17, 32 and 40 and
have exhausted all of my own ideas. Can anyone offer advise to a
Trilinos and MPI newbie?
Many thanks,
Jean-Paul
_______________________________________________
dealii mailing list http://poisson.dealii.org/mailman/listinfo/dealii
