Thanks Jeff, we're working to verify. I don't mind the slower behavior on the trunk; we are only concerned with the stable release series. Enabling thread safety options on the trunk/master is no problem here.
Did the 'more expensive freelists' change make it to the release series as well? Andrew > -----Original Message----- > From: devel [mailto:devel-boun...@open-mpi.org] On Behalf Of Jeff > Squyres (jsquyres) > Sent: Saturday, November 1, 2014 4:49 AM > To: Open MPI Developers List > Subject: Re: [OMPI devel] enable-smp-locks affects PSM performance > > Andrew -- > > A PR went in on the v1.8 branch that removed the locking behavior by > default: > > https://github.com/open-mpi/ompi-release/pull/67 > > Can you confirm that the behavior on the v1.8 nightly tarballs show better > PSM message rate behavior? > > (http://www.open-mpi.org/nightly/v1.8/ -- 135 and beyond have the fix) > > For master, this was actually the exactly intended result: at a previous OMPI > dev meeting, we said "let's turn on threading, and see who screams." You > screamed. :-) > > In a conversation with Nathan on Friday, it looks like many things in > 6ef938de3fa9ca0fed2c5bcb0736f65b0d8803af get more expensive with > threading enabled -- particularly freelists. It's quite possible that we > should > change many of these things to be more expensive only a) if > THREAD_MULTIPLE is active, and/or b) if progress threads are active. > > However, that may not be safe in itself. Perhaps we should have multiple > flavors of these types (like freelists) that are both thread safe and thread > unsafe. So even if we're in a THREAD_MULTIPLE and/or progress threads are > active, if you *know* you've got a data structure that does not need the > added thread-safety protections, you can use the unsafe-but-faster > versions. > > > > > On Oct 30, 2014, at 12:51 PM, Friedley, Andrew > <andrew.fried...@intel.com> wrote: > > > Hi Howard, > > > > No, we do not, just the OMPI default. > > > > This post isn't so much about our packaging requirements, as about default > behavior in upstream Open MPI. We'd like performance to be good by > default for anyone compiling their own Open MPI (and using PSM). > > > > Andrew > > > > From: devel [mailto:devel-boun...@open-mpi.org] On Behalf Of Howard > Pritchard > > Sent: Thursday, October 30, 2014 8:53 AM > > To: Open MPI Developers > > Subject: Re: [OMPI devel] enable-smp-locks affects PSM performance > > > > Hi Andrew, > > > > In your distribution of ompi, do you include versions of ompi to support > different MPI thread safetylevels? In particular, do you include a library > which supports MPI_THREAD_MULTIPLE? > > Just trying to better understand the requirements of you ompi package in > terms of MPI thread safety. > > > > Thanks, > > > > Howard > > > > > > 2014-10-30 8:10 GMT-06:00 Friedley, Andrew > <andrew.fried...@intel.com>: > > Hi, > > > > I'm reporting a performance (message rate 16%, latency 3%) regression > when using PSM that occurred between OMPI v1.6.5 and v1.8.1. I would > guess it affects other networks too, but I haven't tested. The problem stems > from the --enable-smp-locks and --enable-opal-multi-threads options. > > > > --enable-smp-locks defaults to enabled and, on x86, causes a 'lock' prefix > > to > be prepended to ASM instructions used by atomic primitives. Disabling > removes the 'lock' prefix. > > > > In OMPI 1.6.5, --enable-opal-multi-threads defaulted to disabled. When > enabled, OPAL would be compiled with multithreading support, which > included compiling in calls to atomic primitives. Those atomic primitives, in > turn, potentially use a lock prefix (controlled by --enable-smp-locks). > > > > SVN r29891 on the trunk changed the above. --enable-opal-multi-threads > was removed. CPP macros (#if OPAL_ENABLE_MULTI_THREADS) controlling > various calls to atomic primitives were removed, effectively changing the > default behavior to multithreading ON for OPAL. This change was then > carried to the v1.7 branch in r29944, Fixes #3983. > > > > We can use --disable-smp-locks to make the performance regression go > away for the builds we ship, but we'd very much prefer if performance was > good 'out of the box' for people that grab an OMPI tarball and use it with > PSM. > > > > My question is, what's the best way to do that? It seems obvious to just > make --disable-smp-locks the default, but I presume the change was done > on purpose, so I'm looking for community feedback. > > > > Thanks, > > > > Andrew > > _______________________________________________ > > devel mailing list > > de...@open-mpi.org > > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel > > Link to this post: http://www.open- > mpi.org/community/lists/devel/2014/10/16130.php > > > > _______________________________________________ > > devel mailing list > > de...@open-mpi.org > > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel > > Link to this post: http://www.open- > mpi.org/community/lists/devel/2014/10/16133.php > > > -- > Jeff Squyres > jsquy...@cisco.com > For corporate legal information go to: > http://www.cisco.com/web/about/doing_business/legal/cri/ > > _______________________________________________ > devel mailing list > de...@open-mpi.org > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel > Link to this post: http://www.open- > mpi.org/community/lists/devel/2014/11/16141.php
