On Thu, 2007-08-30 at 10:26 -0400, rolf.vandeva...@sun.com wrote:
> Li-Ta Lo wrote:
>
> >On Tue, 2007-08-28 at 10:12 -0600, Brian Barrett wrote:
> >
> >
> >>On Aug 28, 2007, at 9:05 AM, Li-Ta Lo wrote:
> >>
> >>
> >>
> >>>On Mon, 2007-08-27 at 15:10 -0400, Rolf vandeVaart wrote:
> >>>
> >>>
> >>>>We are running into a problem when running on one of our larger SMPs
> >>>>using the latest Open MPI v1.2 branch. We are trying to run a job
> >>>>with np=128 within a single node. We are seeing the following error:
> >>>>
> >>>>"SM failed to send message due to shortage of shared memory."
> >>>>
> >>>>We then increased the allowable maximum size of the shared segment to
> >>>>2Gigabytes-1 which is the maximum allowed on 32-bit application. We
> >>>>used the mca parameter to increase it as shown here.
> >>>>
> >>>>-mca mpool_sm_max_size 2147483647
> >>>>
> >>>>This allowed the program to run to completion. Therefore, we would
> >>>>like to increase the default maximum from 512Mbytes to 2G-1
> >>>>Gigabytes.
> >>>>Does anyone have an objection to this change? Soon we are going to
> >>>>have larger CPU counts and would like to increase the odds that
> >>>>things
> >>>>work "out of the box" on these large SMPs.
> >>>>
> >>>>
> >>>>
> >>>There is a serious problem with the 1.2 branch, it does not allocate
> >>>any SM area for each process at the beginning. SM areas are allocated
> >>>on demand and if some of the processes are more aggressive than the
> >>>others, it will cause starvation. This problem is fixed in the trunk
> >>>by assign at least one SM area for each process. I think this is what
> >>>you saw (starvation) and an increase of max size may not be necessary.
> >>>
> >>>
> >>Although I'm pretty sure this is fixed in the v1.2 branch already.
> >>
> >>
> >>
> >
> >It should never happen for the new code. The only way we can get the
> >message is when MCA_BTL_SM_FIFO_WRITE return rc != OMPI_SUCCESS, but
> >the new MCA_BTL_SM_FIFO_WRITE always return rc = OMPI_SUCCESS
> >
> >#define MCA_BTL_SM_FIFO_WRITE(endpoint_peer,
> >my_smp_rank,peer_smp_rank,hdr,rc) \
> >do { \
> > ompi_fifo_t* fifo; \
> > fifo=&(mca_btl_sm_component.fifo[peer_smp_rank][my_smp_rank]); \
> > \
> > /* thread lock */ \
> > if(opal_using_threads()) \
> > opal_atomic_lock(fifo->head_lock); \
> > /* post fragment */ \
> > while(ompi_fifo_write_to_head(hdr, fifo, \
> > mca_btl_sm_component.sm_mpool) != OMPI_SUCCESS) \
> > opal_progress(); \
> > MCA_BTL_SM_SIGNAL_PEER(endpoint_peer); \
> > rc=OMPI_SUCCESS; \
> > if(opal_using_threads()) \
> > opal_atomic_unlock(fifo->head_lock); \
> >} while(0)
> >
> >Rolf, are you using the really last 1.2 branch?
> >
> >Ollie
> >
> >
> >
> Thanks for all the input. It turns out I was originally *not* using
> the latest 1.2 branch. So, we redid the tests with the latest 1.2.
> And, I am happy to report that we no longer get the"SM failed to
> send message due to shortage of shared memory" error. However,
> now the program hangs. So, it looks like we traded one problem for
> another.
>
Can I see your test code?
Ollie
