Thanks for the valuable input. I'll change to a wait-and-watch approach.
The FAQ on tuning sm says "If the session directory is located on a network
filesystem, the shared memory BTL latency will be extremely high." And the
title is 'Why am I seeing incredibly poor performance...'. So I made the leap
that this configuration must be avoided at all costs...
-Original Message-
From: users-boun...@open-mpi.org [mailto:users-boun...@open-mpi.org] On Behalf
Of David Singleton
Sent: Sunday, November 06, 2011 4:15 PM
To: us...@open-mpi.org
Subject: Re: [OMPI users] EXTERNAL: Re: How to set up state-less node /tmp for
OpenMPI usage
On 11/05/2011 09:11 AM, Blosch, Edwin L wrote:
..
>
> I know where you're coming from, and I probably didn't title the post
> correctly because I wasn't sure what to ask. But I definitely saw it, and
> still see it, as an OpenMPI issue. Having /tmp mounted over NFS on a
> stateless cluster is not a broken configuration, broadly speaking. The
> vendors made those decisions and presumably that's how they do it for other
> customers as well. There are two other (Platform/HP) MPI applications that
> apparently work normally. But OpenMPI doesn't work normally. So it's
> deficient.
>
I'm also concerned that there is a bit of an over-reaction to network
filesystems. Stores to mmap'd files do not instantly turn into filesystem
writes - there are dirty_writeback parameters to control how often
writes occur and its typically 5-20 seconds. Ideally, memory or a local
disk is used for session directories but, in many cases, you just wont
notice a performance hit from network filesystems - we didn't when we
tested session directories on Lustre. If your app is one of those handful
that is slowed by OS jitter at megascale, then you may well notice.
Obviously, its something to test.
For our 1.5 install, I removed Lustre from the list of filesystem types
that generate the warning message about network filesystems. It would be
nice if it was a site choice whether or not to produce that message and
when.
David
___
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users
