By the way: After some time I see the additional function opal_progress() on top of the stack. Also here is what I use: gcc (GCC) 6.3.1 20170306 openmpi 1.10.6-1 trilinos-12.6.1 dealii-8.4.1 and my testcases consit of 4 MPI processes.
Am Dienstag, 14. März 2017 19:29:09 UTC+1 schrieb Pascal Kraft: > > Dear list members, > > I am facing a really weird problem, that I have been struggling with for a > while now. I have written a problem class which, based on other objects, > generates a system matrix, rhs and solution vector object. The > datastructures are Trilinos Block distributed types. When I do this for the > first time it all works perfectly. However the class is part of an > optimization scheme and usually at the second time the object is used > (randomly also later, but this has only happened once or twice) the Solver > does not start. I am checking with MPI-barriers to see if all processes > arrive at the GMRES::solve and they do but somehow not even my own > preconditioners vmult method gets called anymore. The objects (the two > vectors and the system matrix are exactly the same that they have been at > the previous step (only slightly different numbers, but same vectors of > IndexSets for the partition among processors) > > I have debugged this code-segment with Eclipse and the parallel debugger > but don't know what to do with the call stack: > 18 ompi_request_default_wait_all() 7fffddd54b15 > 17 ompi_coll_tuned_barrier_intra_recursivedoubling() 7fffcf9abb5d > 16 PMPI_Barrier() 7fffddd68a9c > 15 Epetra_MpiDistributor::DoPosts() 7fffe4088b4f > 14 Epetra_MpiDistributor::Do() 7fffe4089773 > 13 Epetra_DistObject::DoTransfer() 7fffe400a96a > 12 Epetra_DistObject::Export() 7fffe400b7b7 > 11 int Epetra_FEVector::GlobalAssemble<int>() 7fffe4023d7f > 10 Epetra_FEVector::GlobalAssemble() 7fffe40228e3 > 9 dealii::TrilinosWrappers::MPI::Vector::reinit() trilinos_vector.cc:261 > 7ffff52c937e > 8 dealii::TrilinosWrappers::MPI::BlockVector::reinit() > trilinos_block_vector.cc:191 7ffff4e43bd9 > 7 > dealii::internal::SolverGMRES::TmpVectors<dealii::TrilinosWrappers::MPI::BlockVector>::operator() > > solver_gmres.h:535 4a847d > 6 > dealii::SolverGMRES<dealii::TrilinosWrappers::MPI::BlockVector>::solve<dealii::TrilinosWrappers::BlockSparseMatrix, > > PreconditionerSweeping>() solver_gmres.h:813 4d654a > 5 Waveguide::solve() Waveguide.cpp:1279 48f150 > > The last line (5) here is a function I wrote which calls > SolverGMRES<dealii::TrilinosWrappers::MPI::BlockVector>::solve with my > preconditioner (which works perfecly fine during the previous run. I found > some information online about MPI_Barrier being instable sometimes but I > don't know enough about the inner workings of Trilinos (Epetra) and Dealii > to make a judgment call here. If none can help I will try to provide a code > fragment but I doubt that will be possible (if it really is a racing > condition and I strip away the rather large ammout of code surrounding this > segment, it is unlikely to be reproducible. > > Originally I had used two MPI communicatorsthat were only different in the > numbering of the processes (one for the primal, one for the dual problem) > and created two independend objects of my problem class hich only used > their respective communicator. In that case, the solver had only worked > whenever the numbering of processes was either equal to that of > MPI_COMM_WORLD or exactly the opposite but not for say 1-2-3-4 -> 1-3-2-4 > and gotten stuck in the exact same way. I had thought it might be some > internal use of MPI_COMM_WORLD that was blocking somehow but it also > happens now that I only use one communicator (MPI_COMM_WORLD). > > Thank you in advance for your time, > Pascal Kraft > -- The deal.II project is located at http://www.dealii.org/ For mailing list/forum options, see https://groups.google.com/d/forum/dealii?hl=en --- You received this message because you are subscribed to the Google Groups "deal.II User Group" group. To unsubscribe from this group and stop receiving emails from it, send an email to dealii+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
