Ok. Should be pretty easy to test/simulate to figure out what's
going
on -- e.g., whether it's segv'ing in MPI_INIT or the first MPI_SEND.
On Nov 14, 2008, at 9:19 AM, Ralph Castain wrote:
> Until we do complete the switch, and for systems that do not
support
> the alternate type of shared memory (I believe it is only
Linux?), I
> truly believe we should do something nicer than segv.
>
> Just to clarify: I know the segv case was done with paffinity set,
> and believe both cases were done that way. In the first case, I was
> told that the segv hit when they did MPI_Send, but I did not
> personally verify that claim - it could be that it hit during
> maffinity binding if, as you suggest, we actually touch the page at
> that time.
>
> Ralph
>
>
>
> On Nov 14, 2008, at 7:07 AM, Jeff Squyres wrote:
>
>> It's been a looooong time since I've looked at the sm code; Eugene
>> has looked at it much more in-depth recently than I have. But I'm
>> guessing we *haven't* checked this stuff to abort nicely in such
>> error conditions. We might very well succeed in the mmap but then
>> segv later when the memory isn't actually available. Perhaps we
>> should try to touch every page first to ensure that it's actually
>> there...? (I'm pretty sure we do this when using paffinity to
>> ensure to maffinity bind memory to processors -- perhaps we're not
>> doing that in the !paffinity case?)
>>
>> Additionally, another solution might well be what Tim has long
>> advocated: switch to the other type of shared memory on systems
>> that support auto-pruning it when all processes die, and/or have
>> the orted kill it when all processes die. Then a) we're not
>> dependent on the filesystem free space, and b) we're not writing
>> all the dirty pages to disk when the processes exit.
>>
>>
>>
>> On Nov 14, 2008, at 8:42 AM, Ralph Castain wrote:
>>
>>> Hi Eugene
>>>
>>> I too am interested - I think we need to do something about the
sm
>>> backing file situation as larger core machines are slated to
>>> become more prevalent shortly.
>>>
>>> I appreciate your info on the sizes and controls. One other
>>> question: what happens when there isn't enough memory to support
>>> all this? Are we smart enough to detect this situation? Does the
>>> sm subsystem quietly shut down? Warn and shut down? Segfault?
>>>
>>> I have two examples so far:
>>>
>>> 1. using a ramdisk, /tmp was set to 10MB. OMPI was run on a
single
>>> node, 2ppn, with btl=openib,sm,self. The program started, but
>>> segfaulted on the first MPI_Send. No warnings were printed.
>>>
>>> 2. again with a ramdisk, /tmp was reportedly set to 16MB
>>> (unverified - some uncertainty, could be have been much larger).
>>> OMPI was run on multiple nodes, 16ppn, with btl=openib,sm,self.
>>> The program ran to completion without errors or warning. I don't
>>> know the communication pattern - could be no local comm was
>>> performed, though that sounds doubtful.
>>>
>>> If someone doesn't know, I'll have to dig into the code and
figure
>>> out the response - just hoping that someone can spare me the
pain.
>>>
>>> Thanks
>>> Ralph
>>>
>>>
>>> On Nov 13, 2008, at 3:21 PM, Eugene Loh wrote:
>>>
>>>> Ralph Castain wrote:
>>>>
>>>>> As has frequently been commented upon at one time or another,
>>>>> the shared memory backing file can be quite huge. There used
to
>>>>> be a param for controlling this size, but I can't find it in
>>>>> 1.3 - or at least, the name or method for controlling file
size
>>>>> has morphed into something I don't recognize.
>>>>>
>>>>> Can someone more familiar with that subsystem point me to one
or
>>>>> more params that will allow us to control the size of that
>>>>> file? It is swamping our systems and causing OMPI to segfault.
>>>>
>>>> Sounds like you've already gotten your answers, but I'll add my
>>>> $0.02 anyhow.
>>>>
>>>> The file size is the number of local processes (call it n) times
>>>> mpool_sm_per_peer_size (default 32M), but with a minimum of
>>>> mpool_sm_min_size (default 128M) and a maximum of
>>>> mpool_sm_max_size (default 2G? 256M?). So, you can tweak those
>>>> parameters to control file size.
>>>>
>>>> Another issue is possibly how small a backing file you can get
>>>> away with. That is, just forcing the file to be smaller may not
>>>> be enough since your job may no longer run. The backing file
>>>> seems to be used mainly by:
>>>>
>>>> *) eager-fragment free lists: We start with enough eager
>>>> fragments so that we could have two per connection. So, you
>>>> could bump the sm eager size down if you need to shoehorn a job
>>>> into a very small backing file.
>>>>
>>>> *) large-fragment free lists: We start with 8*n large
>>>> fragments. If this term plagues you, you can bump the sm chunk
>>>> size down or reduce the value of 8 (using
btl_sm_free_list_num, I
>>>> think).
>>>>
>>>> *) FIFOs: The code tries to align a number of things on
pagesize
>>>> boundaries, so you end up with about 3*n*n*pagesize overhead
>>>> here. If this term is causing you problems, you're stuck
(unless
>>>> you modify OMPI).
>>>>
>>>> I'm interested in this subject! :^)
>>>> _______________________________________________
>>>> devel mailing list
>>>> de...@open-mpi.org
>>>> http://www.open-mpi.org/mailman/listinfo.cgi/devel
>>>
>>> _______________________________________________
>>> devel mailing list
>>> de...@open-mpi.org
>>> http://www.open-mpi.org/mailman/listinfo.cgi/devel
>>
>>
>> --
>> Jeff Squyres
>> Cisco Systems
>>
>> _______________________________________________
>> devel mailing list
>> de...@open-mpi.org
>> http://www.open-mpi.org/mailman/listinfo.cgi/devel
>
> _______________________________________________
> devel mailing list
> de...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/devel
--
Jeff Squyres
Cisco Systems
_______________________________________________
devel mailing list
de...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/devel
