On Tue, Feb 7, 2012 at 8:52 PM, Derek Gaston <friedmud at gmail.com> wrote:
> On Mon, Feb 6, 2012 at 11:20 PM, Jed Brown <jedbrown at mcs.anl.gov> wrote: > >> >> Hmm, progress semantics of MPI should ensure completion. Stalling the >> process with gdb should not change anything (assuming you weren't actually >> making changes with gdb). Can you run with MPICH2? >> > > Ok - an update on this. I recompiled my whole stack with mvapich2... and > it still is hanging in the same place: > > #0 0x00002b336a732f40 in PMI_Get_rank () from > /apps/local/mvapich2/1.7/intel-12.1.1/opt/lib/libmpich.so.3 > #1 0x00002b336a6bf453 in MPIDI_CH3I_MRAILI_Cq_poll () from > /apps/local/mvapich2/1.7/intel-12.1.1/opt/lib/libmpich.so.3 > #2 0x00002b336a675818 in MPIDI_CH3I_read_progress () from > /apps/local/mvapich2/1.7/intel-12.1.1/opt/lib/libmpich.so.3 > #3 0x00002b336a67485b in MPIDI_CH3I_Progress () from > /apps/local/mvapich2/1.7/intel-12.1.1/opt/lib/libmpich.so.3 > #4 0x00002b336a6bea96 in MPIC_Wait () from > /apps/local/mvapich2/1.7/intel-12.1.1/opt/lib/libmpich.so.3 > #5 0x00002b336a6be9db in MPIC_Sendrecv () from > /apps/local/mvapich2/1.7/intel-12.1.1/opt/lib/libmpich.so.3 > #6 0x00002b336a6be8aa in MPIC_Sendrecv_ft () from > /apps/local/mvapich2/1.7/intel-12.1.1/opt/lib/libmpich.so.3 > #7 0x00002b336a652db1 in MPIR_Allgather_intra_MV2 () from > /apps/local/mvapich2/1.7/intel-12.1.1/opt/lib/libmpich.so.3 > #8 0x00002b336a652965 in MPIR_Allgather_MV2 () from > /apps/local/mvapich2/1.7/intel-12.1.1/opt/lib/libmpich.so.3 > #9 0x00002b336a651846 in MPIR_Allgather_impl () from > /apps/local/mvapich2/1.7/intel-12.1.1/opt/lib/libmpich.so.3 > #10 0x00002b336a6517b1 in PMPI_Allgather () from > /apps/local/mvapich2/1.7/intel-12.1.1/opt/lib/libmpich.so.3 > #11 0x00000000004a1f23 in PetscLayoutSetUp () > #12 0x000000000054e469 in MatMPIAIJSetPreallocation_MPIAIJ () > #13 0x000000000055584a in MatCreateMPIAIJ () > > It's been hung there for about 35 minutes. > > This particular job has ~100 million DoFs with 512 MPI processes. Any > ideas? > Same question: are you sure every process is there. I will bet $10 there is at least one missing. Matt > Derek > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120207/cec94b65/attachment.htm>
