Re: [OMPI devel] Trunk Commit Heads-up: New Common Shared Memory Component

[Samuel K. Gutierrez]

Code is in (see r23633).  Note: mmap is still the default.

--
Samuel K. Gutierrez
Los Alamos National Laboratory

On Aug 12, 2010, at 11:37 AM, Samuel K. Gutierrez wrote:

Sorry, I should have included the link containing the discussion of  
the plot.


http://www.open-mpi.org/community/lists/devel/2010/06/8078.php

--
Samuel K. Gutierrez
Los Alamos National Laboratory

On Aug 12, 2010, at 11:20 AM, Terry Dontje wrote:

Sorry Rich, I didn't realize there was a graph attached at the end  
of message.  In other words my comments are not applicable because  
I really didn't know you were asking about the graph.  I agree it  
would be nice to know what the graph was plotting.


--td
Terry Dontje wrote:


Graham, Richard L. wrote:


Stupid question:
   What is being plotted, and what are the units ?

Rich

MB of Resident and Shared memory as gotten from top (on linux).   
The values for each of the processes run cases seem to be the same  
between posix, mmap and sysv.


--td

On 8/11/10 3:15 PM, "Samuel K. Gutierrez"  wrote:

Hi Terry,










On Aug 11, 2010, at 12:34 PM, Terry Dontje wrote:


 I've done some minor testing on Linux looking at resident and  
shared memory sizes for np=4, 8 and 16 jobs.  I could not see any  
appreciable differences in sizes in the process between sysv,  
posix or mmap usage in the SM btl.


 So I am still somewhat non-plussed about making this the  
default.  It seems like the biggest gain out of using posix might  
be one doesn't need to worry about the location of the backing  
file.  This seems kind of frivolous to me since there is a  
warning that happens if the backing file is not memory based.


If I'm not mistaken, the warning is only issued if the backing  
files is stored on the following file systems: Lustre, NFS,  
Panasas, and GPFS  (see: opal_path_nfs in opal/util/path.c).   
Based on the performance numbers that Sylvain provided on June  
9th of this year (see attached),  there was a performance  
difference between mmap on /tmp and mmap on a tmpfs-like file  
system (/dev/shm in that particular case).  Using the new POSIX  
component provides us with a portable way to provide similar  
shared memory performance gains without having to worry about  
where the OMPI session directory is rooted.


--
Samuel K. Gutierrez
Los Alamos National Laboratory

[cid:3364459484_11867134]


 I still working on testing the code on Solaris but I don't  
imagine I will see anything that will change my mind.


 --td

 Samuel K. Gutierrez wrote:
Hi Rich,

 It's a modification to the existing common sm component.  The  
modifications do include the addition of a new POSIX shared  
memory facility, however.


 Sam

 On Aug 11, 2010, at 10:05 AM, Graham, Richard L. wrote:


Is this a modification of the existing component, or a new  
component ?


 Rich


 On 8/10/10 10:52 AM, "Samuel K. Gutierrez"    wrote:


 Hi,

 I wanted to give everyone a heads-up about a new POSIX shared  
memory

 component
 that has been in the works for a while now and is ready to be  
pushed

 into the
 trunk.

 http://bitbucket.org/samuelkgutierrez/ompi_posix_sm_new

 Some highlights:
 o New posix component now the new default.
o May address some of the shared memory performance  
issues users

 encounter
   when OMPI's session directories are inadvertently  
placed on a non-

 local
   filesystem.
 o Silent component failover.
o In the default case, if the posix component fails  
initialization,

   mmap will be selected.
 o The sysv component will only be queried for selection if it is
 placed before
the mmap component (for example, -mca mpi_common_sm
 sysv,posix,mmap).  In the
default case, sysv will never be queried/selected.
 o Per some on-list discussion, now unlinking mmaped file in both  
mmap

 and posix
components (see: "System V Shared Memory for Open MPI:  
Request for

 Community
Input and Testing" thread).
 o  Assuming local process homogeneity with respect to all utilized
 shared
 memory facilities. That is, if one local process deems a
 particular shared
 memory facility acceptable, then ALL local processes should be
 able to
 utilize that facility. As it stands, this is an important  
point

 because one
 process dictates to all other local processes which common sm
 component will
 be selected based on its own, local run-time test.
 o Addressed some of George's code reuse concerns.

 If there are no major objections by August 17th, I'll commit the  
code

 after the
 Tuesday morning conference call.

 Thanks!

 --
 Samuel K. Gutierrez
 Los Alamos National Laboratory





 ___
 devel mailing list
 de...@open-mpi.org
 http://www.open-mpi.org/mailman/listinfo.cgi/devel


 ___
 devel mailing list
 de...@open-mpi.org
 http://www.open-mpi.org/mailman/listinfo.cgi/devel


 

[OMPI devel] Question on MCA_BASE_METADATA_PARAM_NONE

[ananda.mudar]

Hi

In the file "mca_base_components_open.c", following code checks for the
components that are checkpointable. If I configure OpenMPI library with
"-enable-cr" option, I was under the assumption that all components will
be checkpointable. However I see that quite a few components are not
checkpointable and that list includes "Shared Memmory (sm)". Do I have
to add any other options to "configure" command so that all components
are checkpointable? Thanks

 186
 /*
 187
  * If the
user asked for a checkpoint enabled run
 188
  * then only
load checkpoint enabled components.
 189
  */
 190
 if(
MCA_BASE_METADATA_PARAM_CHECKPOINT
  & open_only_flags
 ) {
 191
 if(
MCA_BASE_METADATA_PARAM_CHECKPOINT
  & dummy

->data

.param_field
 ) {
 192

opal_output_verbose
 (10, output_id
 ,
 193

"mca: base: components_open: "
 194

"(%s) Component %s is Checkpointable",
 195

type_name
 ,
 196

dummy

->version

.mca_component_name
 );
 197
 }
 198
 else {
 199

opal_output_verbose
 (10, output_id
 ,
 200

"mca: base: components_open: "
 201

"(%s) Component %s is *NOT* Checkpointable - Disabled",
 202

type_name
 ,
 203

dummy

->version

.mca_component_name
 );
 204

opal_list_remove_item

Re: [OMPI devel] Question on MCA_BASE_METADATA_PARAM_NONE

[ananda.mudar]

There is a typo in my previous posting. I have configured OpenMPI
library with "-with-ft=cr".



Thanks

Ananda

From: Ananda Babu Mudar (WT01 - Energy and Utilities)
Sent: Monday, August 23, 2010 11:50 AM
To: 'de...@open-mpi.org'
Subject: Question on MCA_BASE_METADATA_PARAM_NONE



Hi

In the file "mca_base_components_open.c", following code checks for the
components that are checkpointable. If I configure OpenMPI library with
"-enable-cr" option, I was under the assumption that all components will
be checkpointable. However I see that quite a few components are not
checkpointable and that list includes "Shared Memmory (sm)". Do I have
to add any other options to "configure" command so that all components
are checkpointable? Thanks

 186
 /*
 187
  * If the
user asked for a checkpoint enabled run
 188
  * then only
load checkpoint enabled components.
 189
  */
 190
 if(
MCA_BASE_METADATA_PARAM_CHECKPOINT
  & open_only_flags
 ) {
 191
 if(
MCA_BASE_METADATA_PARAM_CHECKPOINT
  & dummy

->data

.param_field
 ) {
 192

opal_output_verbose
 (10, output_id
 ,
 193

"mca: base: components_open: "
 194

"(%s) Component %s is Checkpointable",
 195

type_name
 ,
 196

dummy

->version

.mca_component_name
 );
 197
 }
 198
 else {
 199

opal_output_verbose
 (10, output_id
 ,
 200

"mca: base: components_open: "
 201

"(%s) Component %s is *NOT* Checkpointable - Disabled",
 202

type_name
 ,
 203

dummy

->version

Re: [OMPI devel] Question on MCA_BASE_METADATA_PARAM_NONE

[Joshua Hursey]

When you configure with '--with-ft=cr' this enables the C/R fault tolerance 
frameworks, tools and code paths. One code path is the component selection 
logic you cited below. When you run an application compiled with Open MPI 
passing the '-am ft-enable-cr' or '-am ft-enable-cr-recovery' options this 
activates the logic below to pick only those components that have self 
identified as 'checkpoint ready'. 'checkpoint ready' means different things for 
different frameworks. Some frameworks do not need to do anything (e.g., timer), 
while others require much more work (e.g., BTLs).

There are some components that have not been verified to work well under C/R 
scenarios, and they are not selected when you pass the '-am ' parameters cited 
above. The Shared Memory BTL -is- checkpoint ready, and -will- be selected (on 
the current 1.4, 1.5 and trunk branches).  See the code below (Line 94):
  
https://svn.open-mpi.org/trac/ompi/browser/trunk/ompi/mca/btl/sm/btl_sm_component.c#L94

The shared memory collective module [also called 'sm'] (which is not enabled 
under normal use due to testing - Line 89 in coll_sm_component.c) is -not- 
checkpoint ready (line 77), also due to testing:
  
https://svn.open-mpi.org/trac/ompi/browser/trunk/ompi/mca/coll/sm/coll_sm_component.c#L76

So shared memory communication support has been available for 
checkpoint/restart functionality for a couple years now. The shared memory 
collective has not matured or been tested enough to be active even under 
non-C/R circumstances. Once it is ready, we can consider possibly trying to 
support it under C/R enabled activities.

I hope that clarifies what is going on.

-- Josh

On Aug 23, 2010, at 12:50 PM,   
wrote:

> Hi
>  
> In the file “mca_base_components_open.c”, following code checks for the 
> components that are checkpointable. If I configure OpenMPI library with 
> “—enable-cr” option, I was under the assumption that all components will be 
> checkpointable. However I see that quite a few components are not 
> checkpointable and that list includes “Shared Memmory (sm)”. Do I have to add 
> any other options to “configure” command so that all components are 
> checkpointable? Thanks
>  
>  186/*
>  187 * If the user asked for a checkpoint enabled run
>  188 * then only load checkpoint enabled components.
>  189 */
>  190if( MCA_BASE_METADATA_PARAM_CHECKPOINT & open_only_flags) {
>  191if( MCA_BASE_METADATA_PARAM_CHECKPOINT & 
> dummy->data.param_field) {
>  192opal_output_verbose(10, output_id,
>  193"mca: base: components_open: "
>  194"(%s) Component %s is 
> Checkpointable",
>  195type_name,
>  196
> dummy->version.mca_component_name);
>  197}
>  198else {
>  199opal_output_verbose(10, output_id,
>  200"mca: base: components_open: "
>  201"(%s) Component %s is *NOT* 
> Checkpointable - Disabled",
>  202type_name,
>  203
> dummy->version.mca_component_name);
>  204opal_list_remove_item(&components_found, item);
>  205}
>  206}
>  207}
>  208}
>  
> Thanks
> 
> Ananda
> 
>  
> Ananda B Mudar, PMP
> Senior Technical Architect
> Wipro Technologies
> Please do not print this email unless it is absolutely necessary.
> 
> The information contained in this electronic message and any attachments to 
> this message are intended for the exclusive use of the addressee(s) and may 
> contain proprietary, confidential or privileged information. If you are not 
> the intended recipient, you should not disseminate, distribute or copy this 
> e-mail. Please notify the sender immediately and destroy all copies of this 
> message and any attachments.
> 
> WARNING: Computer viruses can be transmitted via email. The recipient should 
> check this email and any attachments for the presence of viruses. The company 
> accepts no liability for any damage caused by any virus transmitted by this 
> email.
> 
> www.wipro.com
> 
> 


Joshua Hursey
Postdoctoral Research Associate
Oak Ridge National Laboratory
http://www.cs.indiana.edu/~jjhursey

Re: [OMPI devel] 1.5rc5 has been posted

[Larry Baker]

The PGI C compiler complains (issues a warning) for the redefinition  
of the assert macro in opal/mca/memory/ptmalloc2/malloc.c:


Making all in mca/memory/ptmalloc2
make[2]: Entering directory `/home/baker/openmpi-1.5rc5/opal/mca/ 
memory/ptmalloc2'

  CC opal_ptmalloc2_component.lo
  CC opal_ptmalloc2_munmap.lo
  CC malloc.lo
PGC-W-0221-Redefinition of symbol assert (/usr/include/assert.h: 51)
PGC-W-0258-Argument 1 in macro assert is not identical to previous  
definition (/usr/include/assert.h: 51)


FYI.  assert.h is an unusual include file -- it does not use an ifdef  
guard macro in the usual way, but undef's assert if the guard macro is  
defined (NOT if assert is defined, which is the root cause of this  
warning), define's the guard macro, then (re)define's assert() based  
on the current value of NDEBUG.


opal/mca/memory/ptmalloc2/malloc.c did not change from OpenMPI 1.4.2.   
malloc.c include's opal/mca/memory/ptmalloc2/hooks.c, which did change  
in OpenMPI 1.5rc5.  hooks.c indirectly include's  through  
opal/mca/base/mca_base_param.h.  This is where the warning occurs.


malloc.c define's its own assert macro in lines 364-369:

364 #if MALLOC_DEBUG
365 #include 
366 #else
367 #undef assert
368 #define assert(x) ((void)0)
369 #endif

The warning occurs because the definition of assert in line 368 is not  
the same as the definition in :


# define assert(expr)   (__ASSERT_VOID_CAST (0))

However, there is no reason to define assert here -- the only code in  
malloc.c that needs assert is already inside an #if ! MALLOC_DEBUG  
conditional at line 2450.


The fix is to delete lines 364-396 in opal/mca/memory/ptmalloc2/ 
malloc.c and move the #include  to be inside the conditional  
between lines 2459 and 2461:


2459 #else

 #include 

2461 #define check_chunk(A,P)  do_check_chunk(A,P)


Larry Baker
US Geological Survey
650-329-5608
ba...@usgs.gov

On Aug 17, 2010, at 2:18 PM, Jeff Squyres wrote:


We still have one known possible regression:

https://svn.open-mpi.org/trac/ompi/ticket/2530

But we posted rc5 anyway (there's a bunch of stuff that has been  
pending for a while that is now in).  Please test!


http://www.open-mpi.org/software/ompi/v1.5/

--
Jeff Squyres
jsquy...@cisco.com
For corporate legal information go to:
http://www.cisco.com/web/about/doing_business/legal/cri/


___
devel mailing list
de...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/devel

Re: [OMPI devel] 1.5rc5 has been posted

[Larry Baker]

OpenMPI 1.5rc5 make fails for the PGI 10.3 compilers:

make[2]: Entering directory `/home/baker/openmpi-1.5rc5/opal/tools/ 
wrappers'

  CC opal_wrapper.o
  CCLD   opal_wrapper
../../../opal/.libs/libopen-pal.so: undefined reference to  
`pthread_create'

../../../opal/.libs/libopen-pal.so: undefined reference to `assert'
../../../opal/.libs/libopen-pal.so: undefined reference to  
`pthread_mutex_trylock'
../../../opal/.libs/libopen-pal.so: undefined reference to  
`pthread_atfork'
../../../opal/.libs/libopen-pal.so: undefined reference to  
`pthread_join'

make[2]: *** [opal_wrapper] Error 2
make[2]: Leaving directory `/home/baker/openmpi-1.5rc5/opal/tools/ 
wrappers'

make[1]: *** [all-recursive] Error 1
make[1]: Leaving directory `/home/baker/openmpi-1.5rc5/opal'
make: *** [all-recursive] Error 1

I have not had time yet to track this down.  I am at a conference  
today and tomorrow, so I might not have time to look into this until  
Wednesday.


I configured OpenMPI with:

[baker@hydra openmpi-1.5rc5]$ ./configure >configure.log 2>&1-- 
prefix=/opt/pgi/linux86-64/10.3/openmpi-1.5rc5 --with-sge--with- 
wrapper-cflags="-DNO_PGI_OFFSET"--with-wrapper-cxxflags="- 
DNO_PGI_OFFSET"CC="pgcc -m64"CFLAGS="-g -fast -tp amd64 - 
DNO_PGI_OFFSET"CXX="pgcpp -m64"CXXFLAGS="-g -fast -tp amd64 - 
DNO_PGI_OFFSET"FC="pgf90 -m64"FCFLAGS="-g -fast -tp amd64" 
F77="pgf90 -m64"FFLAGS="-g -fast -tp amd64"


The O/S is CentOS 4.5 Linux x86_64:

[baker@hydra ~]$ cat /etc/redhat-release CentOS release 4.5 (Final)
[baker@hydra ~]$ uname -a
Linux hydra.wr.usgs.gov 2.6.9-55.0.2.ELsmp #1 SMP Tue Jun 26 14:14:47  
EDT 2007 x86_64 x86_64 x86_64 GNU/Linux


Larry Baker
US Geological Survey
650-329-5608
ba...@usgs.gov

On Aug 17, 2010, at 2:18 PM, Jeff Squyres wrote:


We still have one known possible regression:

https://svn.open-mpi.org/trac/ompi/ticket/2530

But we posted rc5 anyway (there's a bunch of stuff that has been  
pending for a while that is now in).  Please test!


http://www.open-mpi.org/software/ompi/v1.5/

--
Jeff Squyres
jsquy...@cisco.com
For corporate legal information go to:
http://www.cisco.com/web/about/doing_business/legal/cri/


___
devel mailing list
de...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/devel