On Jun 9, 2010, at 11:57 AM, Jeff Squyres wrote:
Iiinnnnteresting.
This, of course, begs the question of whether we should use sysv
shmem or not. It seems like the order of preference should be:
- sysv
- mmap in a tmpfs
- mmap in a "regular" (but not networked) fs
The big downer, of course, is the whole "what happens if the job
crashes?" issue. With mmap, an rm -rf will clean up any leftover
files (although looking for them in /dev/shm might be a bit non-
obvious). With sysv, you have to use the ipc* commands to look for
and whack any orphan shmem segments.
System V shared memory cleanup is a concern only if a process dies in
between shmat and shmctl IPC_RMID. Shared memory segment cleanup
should happen automagically in most cases, including abnormal process
termination.
--
Samuel K. Gutierrez
Los Alamos National Laboratory
Right now, the orted/hnp won't clean up any left over sysv
segments. This seems like something we should fix.
But even with that, if the orted/hnp is killed, sysv segments can
get let over. Hrm.
On Jun 9, 2010, at 11:58 AM, Sylvain Jeaugey wrote:
As stated at the conf call, I did some performance testing on a 32
cores
node.
So, here is graph showing 500 timings of an allreduce operation
(repeated
15,000 times for good timing) with sysv, mmap on /dev/shm and mmap on
/tmp.
What is shows :
- sysv has the better performance ;
- having the mmap file in /dev/shm is very close to sysv. We only
have
+0.1 us for a complete allreduce operation, but it seems stable.
The noise
is identical to sysv (must be OS noise) ;
- having the mmap file in /tmp (ext3) decreases performance (+0.4 us
compared to /dev/shm) and seems prone to some "other" noise.
Warning : the graph does not start at 0.
Sylvain
On Tue, 27 Apr 2010, Samuel K. Gutierrez wrote:
Hi,
With Jeff and Ralph's help, I have completed a System V shared
memory
component for Open MPI. I have conducted some preliminary tests
on our
systems, but would like to get test results from a broader audience.
As it stands, mmap is the defaul, but System V shared memory can
be activated
using: -mca mpi_common_sm sysv
Repository:
http://bitbucket.org/samuelkgutierrez/ompi_sysv_sm
Input is greatly appreciated!
--
Samuel K. Gutierrez
Los Alamos National Laboratory
_______________________________________________
devel mailing list
de...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/devel
<sm-compared.png>_______________________________________________
devel mailing list
de...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/devel
--
Jeff Squyres
jsquy...@cisco.com
For corporate legal information go to:
http://www.cisco.com/web/about/doing_business/legal/cri/
_______________________________________________
devel mailing list
de...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/devel
